Viewpoint metadata for omnidirectional video

ABSTRACT

Methods are described herein for signaling information regarding different viewpoints in a multi-viewpoint omnidirectional media presentation. In some embodiments, a container file (which may use the ISO Base Media File Format) is generated containing several tracks. The tracks are grouped using a track-group identifier, where each track-group identifier is associated with a different viewpoint. In some embodiments, a manifest (such as an MPEG-DASFI MPD) is generated, where the manifest includes viewpoint identifiers that identify the viewpoint associated with each stream. In some embodiments, metadata included in a container file and/or in a manifest provides information on the position of each viewpoint, the intervals during which each viewpoint is available, transition effects for transitions between viewpoints, and/or recommended projection formats for corresponding field-of-view ranges.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is national stage application under 35 U.S.C.371 of International Application No. PCT/US2019/025784, entitled“VIEWPOINT METADATA FOR OMNIDIRECTIONAL VIDEO”, filed on Apr. 4, 2019,which claims benefit under 35 U.S.C. § 119(e) from U.S. ProvisionalPatent Application No. 62/653,363 (filed Apr. 5, 2018) and U.S.Provisional Patent Application No. 62/675,524 (filed May 23, 2018), bothof which are entitled “Viewpoint Metadata for Omnidirectional Video,”and both of which are incorporated herein by reference in theirentirety.

BACKGROUND

Omnidirectional video or 360° video is a rapidly growing new formatemerging in the media industry. It is enabled by the growingavailability of VR devices and is able to provide the viewer a greatersense of presence. Compared to conventional rectilinear video (2D or3D), 360° video poses a new and difficult set of engineering challengeson video processing and delivery. Enabling comfort and immersive userexperience calls for high video quality and very low latency, while thelarge video size can be an impediment to delivery of 360° video withhigh quality.

ISO Base Media File Format

Within the ISO/IEC 14496 MPEG-4 standard there are several parts thatdefine file formats for the storage of time-based media. These are allbased and derived from the ISO Base Media File Format (ISO BMFF),described in ISO/IEC 14496-12, “Coding of Audio-Visual Objects, Part 12:ISO Base Media File Format”, 2015. The ISO BMFF is a structural,media-independent definition. ISO BMFF contains structural and mediadata information mainly for timed presentations of media data such asaudio, video, etc. There is also support for un-timed data, such asmeta-data at different levels within the file structure. The logicalstructure of the file is of a movie that in turn contains a set oftime-parallel tracks. The time structure of the file is that the trackscontain sequences of samples in time, and those sequences are mappedinto the timeline of the overall movie. ISO BMFF is based in the conceptof box-structured files. A box-structured file consists of a series ofboxes (sometimes called atoms), which have a size and a type. The typesare 32-bit values and usually chosen to be four printable characters,also known a four-character code (4CC). Un-timed data may be containedin a metadata box, at the file level, or attached to the movie box orone of the streams of timed data, called tracks, within the movie.

Dynamic Streaming Over HTTP (DASH).

MPEG Dynamic Adaptive Streaming over HTTP (MPEG-DASH) is a deliveryformat that dynamically adapts to changing network conditions. MPEG-DASHis described in ISO/IEC 23009-1, “Dynamic adaptive streaming over HTTP(DASH), Part 1: Media Presentation Description and Segment Formats”, May2014, Dynamic HTTP streaming calls for various bitrate alternatives ofthe multimedia content to be available at the server. In addition, themultimedia content may consist of several media components (e.g. audio,video, text), each of which may have different characteristics. InMPEG-DASH, these characteristics are described by Media PresentationDescription (MPD).

FIG. 2 illustrates the MPD hierarchical data model. The MPD describesthe sequence of Periods, where a consistent set of encoded versions ofthe media content components does not change during a Period. EachPeriod has a starting time and duration, and is composed of one ormultiple adaptation sets (Adaptation Set).

An adaptation set represents a set of encoded versions of one or severalmedia content components that have a property in common, such as thelanguage, the media type, the picture aspect ratio, the role, theaccessibility, and the rating property. For instance, an adaptation setmay contain different bitrates of the video component of the samemultimedia content. Another adaptation set may contain differentbitrates of the audio component (e.g. lower quality stereo and higherquality surround sound) of the same multimedia content. Each adaptationset usually includes multiple representations.

A representation describes a deliverable encoded version of one orseveral media components, varying from other representations by bitrate,resolution, number of channels or other characteristics. Eachrepresentation consists of one or multiple segments. The attributes ofRepresentation element such as @id, @bandwidth, @qualityRanking, and@dependencyId are used to specify the properties of the associatedrepresentation. Representations may also include sub-representations,which are part of the representation, to describe and extract partialinformation from a representation. Sub-representations may provide theability to access a lower quality version of the representation in whichthey are contained.

A segment is the largest unit of data that can be retrieved with asingle HTTP request. Each segment has a URL, an addressable location ona server, which can be downloaded using HTTP GET or HTTP GET with byteranges.

To use this data model, the DASH client parses the MPD XML document,selects a collection of adaptation sets suitable for its environmentbased on information provided in each of the AdaptationSet elements.Within each adaptation set, the client selects one representation,typically based on the value of a @bandwidth attribute, but also takinginto account client decoding and rendering capabilities. The clientdownloads the initialization segment of the selected representations andthen accesses the content by requesting entire segments or byte rangesof segments. Once the presentation has started, the client continuesconsuming the media content by continuously requesting media segments orparts of media segments and playing content according to the mediapresentation timeline. The client may switch representations taking intoaccount updated information from its environment. The client should playthe content continuously across periods. Once the client is consumingmedia contained in the segments towards the end of the announced mediain the representation, then either the media presentation is terminated,a new period is started, or the MPD is re-fetched.

Descriptors in DASH.

MPEG-DASH uses descriptors to provide application-specific informationabout the media content. Descriptor elements are all structured in thesame way, namely they contain a @schemeIdUri attribute that provides aURI to identify the scheme and an optional attribute @value and anoptional attribute @id. The semantics of the element are specific to thescheme employed. The URI identifying the scheme may be a URN or a URL.The MPD does not provide any specific information on how to use theseelements. It is up to the application that employs DASH formats toinstantiate the description elements with appropriate schemeinformation. DASH applications that use one of these elements firstdefine a scheme identifier in the form of a URI and then define thevalue space for the element when that scheme identifier is used. Ifstructured data is used, then any extension element or attribute may bedefined in a separate namespace. Descriptors may appear at a number oflevels within the MPD. The presence of an element at the MPD level meansthat the element is a child of the MPD element. The presence of anelement at the adaptation set level indicates that the element is achild element of an AdaptationSet element. The presence of an element atthe representation level indicates that the element is a child elementof a Representation element.

Omnidirectional Media Format.

Omnidirectional Media Format (OMAF) is a systems standard developed byMPEG as Part 2 of MPEG-I, a set of standards for coding, representation,storage, and delivery of immersive media. OMAF defines a media formatthat enables omnidirectional media applications and focuses mainly on360° videos, images, audio, and associated timed-metadata. The FinalDraft of International Standard (FDIS) of OMAF was released early 2018and is described in ISO/IEC JTC1/SC29/WG11 N17399 “FDIS 23090-2Omnidirectional Media Format”, February 2018.

As part of Phase 1b of MPEG-I, an extension of OMAF supporting a numberof new features including 3DoF plus motion parallax and support formultiple viewpoints is planned for 2019. The requirements for Phase 1bwere released in February 2018 and are described in ISO/IECJTC1/SC29/WG11 N17331 “MPEG-I Phase 1b Requirements”, February 2018. TheOMAF and MPEG-I Phase 1b requirements describe the following concepts:

-   -   Field-of-view (FoV) is the extent of the observable world in        captured/recorded content or in a physical display device.    -   Viewpoint is the point from which the user views the scene; it        usually corresponds to a camera position. Slight head motion        does not necessarily imply a different viewpoint.    -   Sample is all the data associated with a single time.    -   Track is a timed sequence of related samples in an ISO base        media file. For media data, a track corresponds to a sequence of        images or sampled audio.    -   Box is an object-oriented building block in an ISO based media        file defined by a unique type identifier and length.

SUMMARY

In some embodiments, systems and methods are provided for signalingposition information for one or more viewpoints in an omnidirectionalvideo presentation. In some embodiments, a method includes: receiving amanifest (e.g. an MPEG-DASH MPD) for an omnidirectional videopresentation, the video presentation having at least one omnidirectionalvideo associated with a viewpoint; determining based on the manifestwhether a timed-metadata track of viewpoint position is provided for theviewpoint; and in response to a determination that the timed-metadatatrack is provided, determining the viewpoint position based oninformation in the timed-metadata track.

In some embodiments, determining whether a timed-metadata track ofviewpoint position is provided includes determining whether a flag inthe manifest indicates that the viewpoint position is dynamic.

In some embodiments, the manifest includes coordinates indicating afirst viewpoint position.

In some embodiments, the timed-metadata track is identified in themanifest, and the method further comprises fetching the timed-metadatatrack.

In some embodiments, the timed-metadata track includes viewpointpositions in cartesian coordinates. In other embodiments, thetimed-metadata track includes viewpoint positions in longitude andlatitude coordinates.

In some embodiments, the method further includes displaying to a user auser interface, where the user interface allows a user to select theomnidirectional video based on the viewpoint position of theomnidirectional video. The omnidirectional video is displayed to theuser in response to user selection of the omnidirectional video.

In some embodiments, the omnidirectional video presentation includes atleast a first omnidirectional video and a second omnidirectional video.In such embodiments, the displaying of the user interface may include:displaying the first omnidirectional video to the user; and displaying auser interface element or other indication of the second omnidirectionalvideo at a location in the first omnidirectional video at a locationcorresponding to the location of a viewpoint of the secondomnidirectional video.

Methods are described herein for signaling information regardingdifferent viewpoints in a multi-viewpoint omnidirectional mediapresentation. In some embodiments, a container file (which may use theISO Base Media File Format) is generated containing several tracks. Thetracks are grouped using a track-group identifier, where eachtrack-group identifier is associated with a different viewpoint. In someembodiments, a manifest (such as an MPEG-DASH MPD) is generated, wherethe manifest includes viewpoint identifiers that identify the viewpointassociated with each stream. In some embodiments, metadata included in acontainer file and/or in a manifest provides information on one or moreof the following: the position of each viewpoint, the effective range ofeach viewpoint, the intervals during which each viewpoint is available,transition effects for transitions between viewpoints, and recommendedprojection formats for different field-of-view ranges.

In some embodiments, a method is provide for generating a container file(e.g., an ISO Base Media File Format file). At least first and second360-degree video data is received, the first video data representing aview from a first viewpoint and the second 360-degree video datarepresenting a view from a second viewpoint. A container file isgenerated for at least the first video data and the second video data.In the container file, the first video data is organized into a firstset of tracks and the second video data is organized in a second set oftracks. Each of the tracks in the first set of tracks includes a firsttrack-group identifier associated with the first viewpoint, and each ofthe tracks in the second set of tracks includes a second track-groupidentifier associated with the second viewpoint.

In some such embodiments, each of the tracks in the first set of tracksincludes a respective instance of a viewpoint-group-type box thatcontains the first track-group identifier, and each of the tracks in thesecond set of tracks includes a respective instance of aviewpoint-group-type box that contains the second track-groupidentifier.

In some embodiments, the container file is organized in a hierarchicalbox structure, and the container file includes a viewpoint-list box thatidentifies at least a first viewpoint-information box and a secondviewpoint-information box. The first viewpoint-information box includesat least (i) the first track-group identifier and (ii) an indication oftime intervals for which video from the first viewpoint is available.The second viewpoint-information box includes at least (i) the secondtrack-group identifier and (ii) an indication of time intervals forwhich video from the second viewpoint is available. The indications oftime intervals may be lists of instances of respective viewpointavailability interval boxes.

In some embodiments, the container file is organized in a hierarchicalbox structure, and the container file includes a viewpoint-list boxidentifying at least a first viewpoint-information box and a secondviewpoint-information box. The first viewpoint-information box includesat least (i) the first track-group identifier and (ii) an indication ofa position of the first viewpoint. The second viewpoint-information boxincludes at least (i) the second track-group identifier and (ii) anindication of a position of the second viewpoint. The indications ofposition may include cartesian coordinates or latitude and longitudecoordinates, among other options.

In some embodiments, the container file is organized in a hierarchicalbox structure, and the container file includes a viewpoint-list boxidentifying at least a first viewpoint-information box and a secondviewpoint-information box. The first viewpoint-information box includesat least (i) the first track-group identifier and (ii) an indication ofan effective range of the first viewpoint. The secondviewpoint-information box includes at least (i) the second track-groupidentifier and (ii) an indication of an effective range of the secondviewpoint.

In some embodiments, the container file is organized in a hierarchicalbox structure, the container file includes a transition-effect-list boxidentifying at least one transition-effect box, and eachtransition-effect box includes (i) an identifier of a source viewpoint,(ii) an identifier of a destination viewpoint, and (iii) an identifierof a transition type. The identifier of the transition type may identifya basic transition, a viewpoint path transition, or anauxiliary-information-viewpoint-transition, among other options. In acase of a viewpoint path transition, a path-viewpoint-transition box maybe provided that includes a list of viewpoint identifiers. In the caseof an auxiliary-information-viewpoint-transition, anauxiliary-information-viewpoint-transition box may be provided thatincludes a track identifier.

In some embodiments, the container file is organized in a hierarchicalbox structure including a meta box, and the meta box identifies at leastone recommended-projection-list box. Each recommended-projection-listbox may include information identifying (i) a projection type and (ii) acorresponding field-of-view range for the projection type. Theinformation identifying the field-of-view range may include (i) aminimum horizontal field of view angle; (ii) a maximum horizontal fieldof view angle; (iii) a minimum vertical field of view angle; and (iv) amaximum vertical field of view angle.

In some embodiments, a method is provided for generating a manifest,such as an MPEG-DASH MPD. At least first 360-degree video datarepresenting a view from a first viewpoint and second 360-degree videodata representing a view from a second viewpoint are received. Amanifest is generated. In the manifest, at least one stream in a firstset of streams is identified, each stream in the first set representingat least a portion of the first video data. At least one stream in asecond set of streams is also identified in the manifest, each stream inthe second set representing at least a portion of the second video data.Each of the streams in the first set is associated in the manifest witha first viewpoint identifier, and each of the streams in the second setis associated in the manifest with a second viewpoint identifier.

In some embodiments, each of the streams in the first set is associatedin the manifest with a respective adaptation set that has the firstviewpoint identifier as an attribute; and each of the streams in thesecond set is associated in the manifest with a respective adaptationset that has the second viewpoint identifier as an attribute.

In some embodiments, each of the streams in the first set is associatedin the manifest with a respective adaptation set that has the firstviewpoint identifier in a first descriptor; and each of the streams inthe second set is associated in the manifest with a respectiveadaptation set that has the second viewpoint identifier in a seconddescriptor.

In some embodiments, the manifest further includes an attributeindicating an effective range for each of the viewpoints. In someembodiments, the manifest further includes an attribute indicating aposition for each of the viewpoints. The attribute indicating positionmay include Cartesian coordinates or latitude and longitude coordinates.

In some embodiments, the manifest further includes, for each viewpoint,information indicating at least one time period during which video forthe respective viewpoint is available.

In some embodiments, of a method for generating a manifest, the firstvideo data and second video data are received in a container file inwhich: the first video data is organized into a first set of tracks andthe second video data is organized in a second set of tracks; each ofthe tracks in the first set of tracks includes a first track-groupidentifier associated with the first viewpoint; and each of the tracksin the second set of tracks includes a second track-group identifierassociated with the second viewpoint. The viewpoint identifiers used inthe manifest may be equal to the respective track-group identifiers inthe container file.

Some embodiments may be implemented by a client device, such as a deviceequipped with a head-mounted display or other display device for360-degree video. In some such methods, a manifest identifying aplurality of 360-degree video streams is received, where the manifestincludes information identifying a viewpoint location of each respectivestream. A first video stream identified in the manifest is retrieved anddisplayed. A user interface element indicating the viewpoint location ofa second video stream identified in the manifest is overlaid on thedisplay of the first video stream. In response to selection of the userinterface element, the second video stream is retrieved and displayed.

In some such embodiments, the manifest further includes informationidentifying an effective range of at least one of the identifiedstreams, and the client further displays an indication of the effectiverange.

In some embodiments, the manifest further includes in formationidentifying a period of availability of the second video stream, and theuser interface element is displayed only during the period ofavailability.

In some embodiments, the manifest further includes informationidentifying a transition type for a transition from the first videostream to the second video stream. In response to selection of the userinterface element, the client presents a transition having theidentified transition type, and the second video stream is displayedafter the presentation of the transition.

In some embodiments, the manifest further includes informationidentifying a location of at least one virtual viewpoint. In response toselection of the virtual viewpoint, the client synthesizes a view fromthe virtual viewpoint and displays the synthesized view. One or moresynthesized views may be used in a transition.

Methods are further described for selecting a projection format. In someembodiments, a client receives a manifest that identifies a plurality of360-degree video streams. The manifest includes information identifyinga respective projection format of each of the video streams. Themanifest further includes information identifying a respective range offield-of-view sizes for each of the projection formats. The clientdetermines a field-of-view size for display. The client then selects atleast one of the video streams such that the determined field-of-viewsize is within the identified range of field-of-view sizes for theprojection format of the selected video streams. The client retrieves atleast one of the selected video streams and displays the retrieved videostream with the determined field-of-view size.

Further encompassed in the present disclosure are systems comprising aprocessor and a non-transitory computer-readable medium storinginstructions operative when executed on the processor to perform any ofthe methods described herein. Also encompassed in the present disclosureare non-transitory computer-readable storage media storing one or morecontainer files or manifests generated using the methods disclosedherein.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is a system diagram illustrating an example communicationssystem in which one or more disclosed embodiments may be implemented.

FIG. 1B is a system diagram illustrating an example wirelesstransmit/receive unit (WTRU) that may be used within the communicationssystem illustrated in FIG. 1A according to an embodiment.

FIG. 2 illustrates an MPD hierarchical data model.

FIG. 3 illustrates a user interface (UI) showing available viewpoints ina presentation example.

FIG. 4 illustrates another UI design example to indicate multipleviewpoints by overlaying the available viewpoint icons on an HMDdisplay.

FIG. 5 illustrates an example of viewpoints with different effectiveranges.

FIG. 6 illustrates changes in viewpoint availability over time.

FIG. 7 illustrates an example of real viewpoints and intermediatevirtual viewpoints.

FIG. 8 illustrates mapping of pixels from a reference viewpoint to atarget virtual viewpoint.

EXAMPLE NETWORKS FOR IMPLEMENTATION OF THE EMBODIMENTS

FIG. 1A is a diagram illustrating an example communications system 100in which one or more disclosed embodiments may be implemented. Thecommunications system 100 may be a multiple access system that providescontent, such as voice, data, video, messaging, broadcast, etc., tomultiple wireless users. The communications system 100 may enablemultiple wireless users to access such content through the sharing ofsystem resources, including wireless bandwidth. For example, thecommunications systems 100 may employ one or more channel accessmethods, such as code division multiple access (CDMA), time divisionmultiple access (TDMA), frequency division multiple access (FDMA),orthogonal FDMA (OFDMA), single-carrier FDMA (SC-FDMA), zero-tailunique-word DFT-Spread OFDM (ZT UW DTS-s OFDM), unique word OFDM(UW-OFDM), resource block-filtered OFDM, filter bank multicarrier(FBMC), and the like.

As shown in FIG. 1A, the communications system 100 may include wirelesstransmit/receive units (WTRUs) 102 a, 102 b, 102 c, 102 d, a RAN104/113, a CN 106/115, a public switched telephone network (PSTN) 108,the Internet 110, and other networks 112, though it will be appreciatedthat the disclosed embodiments contemplate any number of WTRUs, basestations, networks, and/or network elements. Each of the WTRUs 102 a,102 b, 102 c, 102 d may be any type of device configured to operateand/or communicate in a wireless environment. By way of example, theWTRUs 102 a, 102 b, 102 c, 102 d, any of which may be referred to as a“station” and/or a “STA”, may be configured to transmit and/or receivewireless signals and may include a user equipment (UE), a mobilestation, a fixed or mobile subscriber unit, a subscription-based unit, apager, a cellular telephone, a personal digital assistant (PDA), asmartphone, a laptop, a netbook, a personal computer, a wireless sensor,a hotspot or Mi-Fi device, an Internet of Things (IoT) device, a watchor other wearable, a head-mounted display (HMD), a vehicle, a drone, amedical device and applications (e.g., remote surgery), an industrialdevice and applications (e.g., a robot and/or other wireless devicesoperating in an industrial and/or an automated processing chaincontexts), a consumer electronics device, a device operating oncommercial and/or industrial wireless networks, and the like. Any of theWTRUs 102 a, 102 b, 102 c and 102 d may be interchangeably referred toas a UE.

The communications systems 100 may also include a base station 114 aand/or a base station 114 b. Each of the base stations 114 a, 114 b maybe any type of device configured to wirelessly interface with at leastone of the WTRUs 102 a, 102 b, 102 c, 102 d to facilitate access to oneor more communication networks, such as the CN 106/115, the Internet110, and/or the other networks 112. By way of example, the base stations114 a, 114 b may be a base transceiver station (BTS), a Node-B, an eNodeB, a Home Node B, a Home eNode B, a gNB, a NR NodeB, a site controller,an access point (AP), a wireless router, and the like. While the basestations 114 a, 114 b are each depicted as a single element, it will beappreciated that the base stations 114 a, 114 b may include any numberof interconnected base stations and/or network elements.

The base station 114 a may be part of the RAN 104/113, which may alsoinclude other base stations and/or network elements (not shown), such asa base station controller (BSC), a radio network controller (RNC), relaynodes, etc. The base station 114 a and/or the base station 114 b may beconfigured to transmit and/or receive wireless signals on one or morecarrier frequencies, which may be referred to as a cell (not shown).These frequencies may be in licensed spectrum, unlicensed spectrum, or acombination of licensed and unlicensed spectrum. A cell may providecoverage for a wireless service to a specific geographical area that maybe relatively fixed or that may change over time. The cell may furtherbe divided into cell sectors. For example, the cell associated with thebase station 114 a may be divided into three sectors. Thus, in oneembodiment, the base station 114 a may include three transceivers, i.e.,one for each sector of the cell. In an embodiment, the base station 114a may employ multiple-input multiple output (MIMO) technology and mayutilize multiple transceivers for each sector of the cell. For example,beamforming may be used to transmit and/or receive signals in desiredspatial directions.

The base stations 114 a, 114 b may communicate with one or more of theWTRUs 102 a, 102 b, 102 c, 102 d over an air interface 116, which may beany suitable wireless communication link (e.g., radio frequency (RF),microwave, centimeter wave, micrometer wave, infrared (IR), ultraviolet(UV), visible light, etc.). The air interface 116 may be establishedusing any suitable radio access technology (RAT).

More specifically, as noted above, the communications system 100 may bea multiple access system and may employ one or more channel accessschemes, such as CDMA, TDMA, FDMA, OFDMA, SC-FDMA, and the like. Forexample, the base station 114 a in the RAN 104/113 and the WTRUs 102 a,102 b, 102 c may implement a radio technology such as Universal MobileTelecommunications System (UMTS) Terrestrial Radio Access (UTRA), whichmay establish the air interface 115/116/117 using wideband CDMA (WCDMA).WCDMA may include communication protocols such as High-Speed PacketAccess (HSPA) and/or Evolved HSPA (HSPA+). HSPA may include High-SpeedDownlink (DL) Packet Access (HSDPA) and/or High-Speed UL Packet Access(HSUPA).

In an embodiment, the base station 114 a and the WTRUs 102 a, 102 b, 102c may implement a radio technology such as Evolved UMTS TerrestrialRadio Access (E-UTRA), which may establish the air interface 116 usingLong Term Evolution (LTE) and/or LTE-Advanced (LTE-A) and/orLTE-Advanced Pro (LTE-A Pro).

In an embodiment, the base station 114 a and the WTRUs 102 a, 102 b, 102c may implement a radio technology such as NR Radio Access, which mayestablish the air interface 116 using New Radio (NR).

In an embodiment, the base station 114 a and the WTRUs 102 a, 102 b, 102c may implement multiple radio access technologies. For example, thebase station 114 a and the WTRUs 102 a, 102 b, 102 c may implement LTEradio access and NR radio access together, for instance using dualconnectivity (DC) principles. Thus, the air interface utilized by WTRUs102 a, 102 b, 102 c may be characterized by multiple types of radioaccess technologies and/or transmissions sent to/from multiple types ofbase stations (e.g., a eNB and a gNB).

In other embodiments, the base station 114 a and the WTRUs 102 a, 102 b,102 c may implement radio technologies such as IEEE 802.11 (i.e.,Wireless Fidelity (WiFi), IEEE 802.16 (i.e., Worldwide Interoperabilityfor Microwave Access (WiMAX)), CDMA2000, CDMA2000 1×, CDMA2000 EV-DO,Interim Standard 2000 (IS-2000), Interim Standard 95 (IS-95), InterimStandard 856 (IS-856), Global System for Mobile communications (GSM),Enhanced Data rates for GSM Evolution (EDGE), GSM EDGE (GERAN), and thelike.

The base station 114 b in FIG. 1A may be a wireless router, Home Node B,Home eNode B, or access point, for example, and may utilize any suitableRAT for facilitating wireless connectivity in a localized area, such asa place of business, a home, a vehicle, a campus, an industrialfacility, an air corridor (e.g., for use by drones), a roadway, and thelike. In one embodiment, the base station 114 b and the WTRUs 102 c, 102d may implement a radio technology such as IEEE 802.11 to establish awireless local area network (WLAN). In an embodiment, the base station114 b and the WTRUs 102 c, 102 d may implement a radio technology suchas IEEE 802.15 to establish a wireless personal area network (WPAN). Inyet another embodiment, the base station 114 b and the WTRUs 102 c, 102d may utilize a cellular-based RAT (e.g., WCDMA, CDMA2000, GSM, LTE,LTE-A, LTE-A Pro, NR etc.) to establish a picocell or femtocell. Asshown in FIG. 1A, the base station 114 b may have a direct connection tothe Internet 110. Thus, the base station 114 b may not be required toaccess the Internet 110 via the CN 106/115.

The RAN 104/113 may be in communication with the CN 106/115, which maybe any type of network configured to provide voice, data, applications,and/or voice over internet protocol (VoIP) services to one or more ofthe WTRUs 102 a, 102 b, 102 c, 102 d. The data may have varying qualityof service (QoS) requirements, such as differing throughputrequirements, latency requirements, error tolerance requirements,reliability requirements, data throughput requirements, mobilityrequirements, and the like. The CN 106/115 may provide call control,billing services, mobile location-based services, pre-paid calling,Internet connectivity, video distribution, etc., and/or performhigh-level security functions, such as user authentication. Although notshown in FIG. 1A, it will be appreciated that the RAN 104/113 and/or theCN 106/115 may be in direct or indirect communication with other RANsthat employ the same RAT as the RAN 104/113 or a different RAT. Forexample, in addition to being connected to the RAN 104/113, which may beutilizing a NR radio technology, the CN 106/115 may also be incommunication with another RAN (not shown) employing a GSM, UMTS, CDMA2000, WiMAX, E-UTRA, or WiFi radio technology.

The CN 106/115 may also serve as a gateway for the WTRUs 102 a, 102 b,102 c, 102 d to access the PSTN 108, the Internet 110, and/or the othernetworks 112. The PSTN 108 may include circuit-switched telephonenetworks that provide plain old telephone service (POTS). The Internet110 may include a global system of interconnected computer networks anddevices that use common communication protocols, such as thetransmission control protocol (TCP), user datagram protocol (UDP) and/orthe internet protocol (IP) in the TCP/IP internet protocol suite. Thenetworks 112 may include wired and/or wireless communications networksowned and/or operated by other service providers. For example, thenetworks 112 may include another CN connected to one or more RANs, whichmay employ the same RAT as the RAN 104/113 or a different RAT.

Some or all of the WTRUs 102 a, 102 b, 102 c, 102 d in thecommunications system 100 may include multi-mode capabilities (e.g., theWTRUs 102 a, 102 b, 102 c, 102 d may include multiple transceivers forcommunicating with different wireless networks over different wirelesslinks). For example, the WTRU 102 c shown in FIG. 1A may be configuredto communicate with the base station 114 a, which may employ acellular-based radio technology, and with the base station 114 b, whichmay employ an IEEE 802 radio technology.

FIG. 1B is a system diagram illustrating an example WTRU 102. As shownin FIG. 1B, the WTRU 102 may include a processor 118, a transceiver 120,a transmit/receive element 122, a speaker/microphone 124, a keypad 126,a display/touchpad 128, non-removable memory 130, removable memory 132,a power source 134, a global positioning system (GPS) chipset 136,and/or other peripherals 138, among others. It will be appreciated thatthe WTRU 102 may include any sub-combination of the foregoing elementswhile remaining consistent with an embodiment.

The processor 118 may be a general purpose processor, a special purposeprocessor, a conventional processor, a digital signal processor (DSP), aplurality of microprocessors, one or more microprocessors in associationwith a DSP core, a controller, a microcontroller, Application SpecificIntegrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs)circuits, any other type of integrated circuit (IC), a state machine,and the like. The processor 118 may perform signal coding, dataprocessing, power control, input/output processing, and/or any otherfunctionality that enables the WTRU 102 to operate in a wirelessenvironment. The processor 118 may be coupled to the transceiver 120,which may be coupled to the transmit/receive element 122. While FIG. 1Bdepicts the processor 118 and the transceiver 120 as separatecomponents, it will be appreciated that the processor 118 and thetransceiver 120 may be integrated together in an electronic package orchip.

The transmit/receive element 122 may be configured to transmit signalsto, or receive signals from, a base station (e.g., the base station 114a) over the air interface 116. For example, in one embodiment, thetransmit/receive element 122 may be an antenna configured to transmitand/or receive RF signals. In an embodiment, the transmit/receiveelement 122 may be an emitter/detector configured to transmit and/orreceive IR, UV, or visible light signals, for example. In yet anotherembodiment, the transmit/receive element 122 may be configured totransmit and/or receive both RF and light signals. It will beappreciated that the transmit/receive element 122 may be configured totransmit and/or receive any combination of wireless signals.

Although the transmit/receive element 122 is depicted in FIG. 1B as asingle element, the WTRU 102 may include any number of transmit/receiveelements 122. More specifically, the WTRU 102 may employ MIMOtechnology. Thus, in one embodiment, the WTRU 102 may include two ormore transmit/receive elements 122 (e.g., multiple antennas) fortransmitting and receiving wireless signals over the air interface 116.

The transceiver 120 may be configured to modulate the signals that areto be transmitted by the transmit/receive element 122 and to demodulatethe signals that are received by the transmit/receive element 122. Asnoted above, the WTRU 102 may have multi-mode capabilities. Thus, thetransceiver 120 may include multiple transceivers for enabling the WTRU102 to communicate via multiple RATs, such as NR and IEEE 802.11, forexample.

The processor 118 of the WTRU 102 may be coupled to, and may receiveuser input data from, the speaker/microphone 124, the keypad 126, and/orthe display/touchpad 128 (e.g., a liquid crystal display (LCD) displayunit or organic light-emitting diode (OLED) display unit). The processor118 may also output user data to the speaker/microphone 124, the keypad126, and/or the display/touchpad 128. In addition, the processor 118 mayaccess information from, and store data in, any type of suitable memory,such as the non-removable memory 130 and/or the removable memory 132.The non-removable memory 130 may include random-access memory (RAM),read-only memory (ROM), a hard disk, or any other type of memory storagedevice. The removable memory 132 may include a subscriber identitymodule (SIM) card, a memory stick, a secure digital (SD) memory card,and the like. In other embodiments, the processor 118 may accessinformation from, and store data in, memory that is not physicallylocated on the WTRU 102, such as on a server or a home computer (notshown).

The processor 118 may receive power from the power source 134, and maybe configured to distribute and/or control the power to the othercomponents in the WTRU 102. The power source 134 may be any suitabledevice for powering the WTRU 102. For example, the power source 134 mayinclude one or more dry cell batteries (e.g., nickel-cadmium (NiCd),nickel-zinc (NiZn), nickel metal hydride (NiMH), lithium-ion (Li-ion),etc.), solar cells, fuel cells, and the like.

The processor 118 may also be coupled to the GPS chipset 136, which maybe configured to provide location information (e.g., longitude andlatitude) regarding the current location of the WTRU 102. In additionto, or in lieu of, the information from the GPS chipset 136, the WTRU102 may receive location information over the air interface 116 from abase station (e.g., base stations 114 a, 114 b) and/or determine itslocation based on the timing of the signals being received from two ormore nearby base stations. It will be appreciated that the WTRU 102 mayacquire location information by way of any suitablelocation-determination method while remaining consistent with anembodiment.

The processor 118 may further be coupled to other peripherals 138, whichmay include one or more software and/or hardware modules that provideadditional features, functionality and/or wired or wirelessconnectivity. For example, the peripherals 138 may include anaccelerometer, an e-compass, a satellite transceiver, a digital camera(for photographs and/or video), a universal serial bus (USB) port, avibration device, a television transceiver, a hands free headset, aBluetooth® module, a frequency modulated (FM) radio unit, a digitalmusic player, a media player, a video game player module, an Internetbrowser, a Virtual Reality and/or Augmented Reality (VR/AR) device, anactivity tracker, and the like. The peripherals 138 may include one ormore sensors, the sensors may be one or more of a gyroscope, anaccelerometer, a hall effect sensor, a magnetometer, an orientationsensor, a proximity sensor, a temperature sensor, a time sensor; ageolocation sensor; an altimeter, a light sensor, a touch sensor, amagnetometer, a barometer, a gesture sensor, a biometric sensor, and/ora humidity sensor.

The WTRU 102 may include a full duplex radio for which transmission andreception of some or all of the signals (e.g., associated withparticular subframes for both the UL (e.g., for transmission) anddownlink (e.g., for reception) may be concurrent and/or simultaneous.The full duplex radio may include an interference management unit toreduce and or substantially eliminate self-interference via eitherhardware (e.g., a choke) or signal processing via a processor (e.g., aseparate processor (not shown) or via processor 118). In an embodiment,the WRTU 102 may include a half-duplex radio for which transmission andreception of some or all of the signals (e.g., associated withparticular subframes for either the UL (e.g., for transmission) or thedownlink (e.g., for reception)).

Although the WTRU is described in FIGS. 1A-1B as a wireless terminal, itis contemplated that in certain representative embodiments that such aterminal may use (e.g., temporarily or permanently) wired communicationinterfaces with the communication network.

In representative embodiments, the other network 112 may be a WLAN.

A WLAN in Infrastructure Basic Service Set (BSS) mode may have an AccessPoint (AP) for the BSS and one or more stations (STAs) associated withthe AP. The AP may have an access or an interface to a DistributionSystem (DS) or another type of wired/wireless network that carriestraffic in to and/or out of the BSS. Traffic to STAs that originatesfrom outside the BSS may arrive through the AP and may be delivered tothe STAs. Traffic originating from STAs to destinations outside the BSSmay be sent to the AP to be delivered to respective destinations.Traffic between STAs within the BSS may be sent through the AP, forexample, where the source STA may send traffic to the AP and the AP maydeliver the traffic to the destination STA. The traffic between STAswithin a BSS may be considered and/or referred to as peer-to-peertraffic. The peer-to-peer traffic may be sent between (e.g., directlybetween) the source and destination STAs with a direct link setup (DLS).In certain representative embodiments, the DLS may use an 802.11e DLS oran 802.11z tunneled DLS (TDLS). A WLAN using an Independent BSS (IBSS)mode may not have an AP, and the STAs (e.g., all of the STAs) within orusing the IBSS may communicate directly with each other. The IBSS modeof communication may sometimes be referred to herein as an “ad-hoc” modeof communication.

When using the 802.11ac infrastructure mode of operation or a similarmode of operations, the AP may transmit a beacon on a fixed channel,such as a primary channel. The primary channel may be a fixed width(e.g., 20 MHz wide bandwidth) or a dynamically set width via signaling.The primary channel may be the operating channel of the BSS and may beused by the STAs to establish a connection with the AP. In certainrepresentative embodiments, Carrier Sense Multiple Access with CollisionAvoidance (CSMA/CA) may be implemented, for example in in 802.11systems. For CSMA/CA, the STAs (e.g., every STA), including the AP, maysense the primary channel. If the primary channel is sensed/detectedand/or determined to be busy by a particular STA, the particular STA mayback off. One STA (e.g., only one station) may transmit at any giventime in a given BSS.

High Throughput (HT) STAs may use a 40 MHz wide channel forcommunication, for example, via a combination of the primary 20 MHzchannel with an adjacent or nonadjacent 20 MHz channel to form a 40 MHzwide channel.

Very High Throughput (VHT) STAs may support 20 MHz, 40 MHz, 80 MHz,and/or 160 MHz wide channels. The 40 MHz, and/or 80 MHz, channels may beformed by combining contiguous 20 MHz channels. A 160 MHz channel may beformed by combining 8 contiguous 20 MHz channels, or by combining twonon-contiguous 80 MHz channels, which may be referred to as an 80+80configuration. For the 80+80 configuration, the data, after channelencoding, may be passed through a segment parser that may divide thedata into two streams. Inverse Fast Fourier Transform (IFFT) processing,and time domain processing, may be done on each stream separately. Thestreams may be mapped on to the two 80 MHz channels, and the data may betransmitted by a transmitting STA. At the receiver of the receiving STA,the above described operation for the 80+80 configuration may bereversed, and the combined data may be sent to the Medium Access Control(MAC).

Sub 1 GHz modes of operation are supported by 802.11af and 802.11ah. Thechannel operating bandwidths, and carriers, are reduced in 802.11af and802.11ah relative to those used in 802.11n, and 802.11ac. 802.11afsupports 5 MHz, 10 MHz and 20 MHz bandwidths in the TV White Space(TVWS) spectrum, and 802.11ah supports 1 MHz, 2 MHz, 4 MHz, 8 MHz, and16 MHz bandwidths using non-TVWS spectrum. According to a representativeembodiment, 802.11ah may support Meter Type Control/Machine-TypeCommunications, such as MTC devices in a macro coverage area. MTCdevices may have certain capabilities, for example, limited capabilitiesincluding support for (e.g., only support for) certain and/or limitedbandwidths. The MTC devices may include a battery with a battery lifeabove a threshold (e.g., to maintain a very long battery life).

WLAN systems, which may support multiple channels, and channelbandwidths, such as 802.11n, 802.11ac, 802.11af, and 802.11ah, include achannel which may be designated as the primary channel. The primarychannel may have a bandwidth equal to the largest common operatingbandwidth supported by all STAs in the BSS. The bandwidth of the primarychannel may be set and/or limited by a STA, from among all STAs inoperating in a BSS, which supports the smallest bandwidth operatingmode. In the example of 802.11ah, the primary channel may be 1 MHz widefor STAs (e.g., MTC type devices) that support (e.g., only support) a 1MHz mode, even if the AP, and other STAs in the BSS support 2 MHz, 4MHz, 8 MHz, 16 MHz, and/or other channel bandwidth operating modes.Carrier sensing and/or Network Allocation Vector (NAV) settings maydepend on the status of the primary channel. If the primary channel isbusy, for example, due to a STA (which supports only a 1 MHz operatingmode), transmitting to the AP, the entire available frequency bands maybe considered busy even though a majority of the frequency bands remainsidle and may be available.

In the United States, the available frequency bands, which may be usedby 802.11ah, are from 902 MHz to 928 MHz. In Korea, the availablefrequency bands are from 917.5 MHz to 923.5 MHz. In Japan, the availablefrequency bands are from 916.5 MHz to 927.5 MHz. The total bandwidthavailable for 802.11ah is 6 MHz to 26 MHz depending on the country code.

In view of FIGS. 1A-1B, and the corresponding description of FIGS.1A-1B, one or more, or all, of the functions described herein withregard to one or more of: WTRU 102 a-d, Base Station 114 a-b, eNode-B160 a-c, MME 162, SGW 164, PGW 166, gNB 180 a-c, AMF 182 a-b, UPF 184a-b, SMF 183 a-b, DN 185 a-b, and/or any other device(s) describedherein, may be performed by one or more emulation devices (not shown).The emulation devices may be one or more devices configured to emulateone or more, or all, of the functions described herein. For example, theemulation devices may be used to test other devices and/or to simulatenetwork and/or WTRU functions.

The emulation devices may be designed to implement one or more tests ofother devices in a lab environment and/or in an operator networkenvironment. For example, the one or more emulation devices may performthe one or more, or all, functions while being fully or partiallyimplemented and/or deployed as part of a wired and/or wirelesscommunication network in order to test other devices within thecommunication network. The one or more emulation devices may perform theone or more, or all, functions while being temporarilyimplemented/deployed as part of a wired and/or wireless communicationnetwork. The emulation device may be directly coupled to another devicefor purposes of testing and/or may performing testing using over-the-airwireless communications.

The one or more emulation devices may perform the one or more, includingall, functions while not being implemented/deployed as part of a wiredand/or wireless communication network. For example, the emulationdevices may be utilized in a testing scenario in a testing laboratoryand/or a non-deployed (e.g., testing) wired and/or wirelesscommunication network in order to implement testing of one or morecomponents. The one or more emulation devices may be test equipment.Direct RF coupling and/or wireless communications via RF circuitry(e.g., which may include one or more antennas) may be used by theemulation devices to transmit and/or receive data.

DETAILED DESCRIPTION

Issues Addressed in Some Embodiments.

An omnidirectional video provides a 360-degree experience that enablesthe viewer to watch the video in all directions around a central viewingposition. However, the viewer has generally been limited to a singleviewpoint and is not able to navigate the scene by changing theirviewpoint. For large-scale events such as the Olympic Games openingceremony, NFL or NBA tournaments, carnival parades, etc., a single 360°video camera is not enough to capture the entire scene. A more enhancedexperience can be provided by capturing the scene from multipleviewpoints and providing the user with the ability to switch betweendifferent viewpoints while watching the video. FIG. 3 shows a userinterface that may be presented to the user in some embodiments toindicate available viewpoints. In this example, the user interfacedisplays an overhead view of a venue and provides indications oflocations of available viewpoints. In this case, viewpoint 302 is theactive viewpoint (the viewpoint from which the user is currentlyexperiencing the presentation) and is displayed in a highlightedfashion. Other viewpoints, such as viewpoints 304, 306, 308, 310, 312,314, 316, may be displayed to indicate their availability, but they arenot currently selected by the user.

During playback, a user interface such as that illustrated in FIG. 3 maybe overlaid over the rendered frame at one of the four corners forexample, and the user can select a different viewpoint using a userinput device such as a touch screen or an HMD controller. A viewpointswitch is then triggered and the user's view is transitioned so thatframes from the target viewpoint are rendered on the display. In someembodiments, a transition effect (e.g., fade-out/fade-in) accompaniesthe transition between viewpoints.

FIG. 4 illustrates another user design example in which the location ofavailable viewpoints is indicated using icons displayed as overlays oncontent 400 displayed on a head-mounted display. The position of eachviewpoint icon in the users view corresponds to the spatial position ofan available viewpoint. In the example of FIG. 4 , icons 406, 414 may bedisplayed to correspond to the viewpoints 306, 314, respectively, ofFIG. 3 . The viewpoint icons may be rendered with the correct deptheffect to allow the user to perceive each viewpoint position in threedimensional space within the scene. For example, an icon 416(corresponding to viewpoint position 316) may be displayed at a largersize than icons 406, 414 to indicate that the viewpoint corresponding toicon 416 is closer to the current viewpoint. The user can select aviewpoint icon in order to switch the user's view of the rendered sceneto the associated viewpoint.

In example embodiments, to enable support for multiple viewpoints,information about the available viewpoints is signaled to the player(which may, for example, be an omnidirectional media player equippedwith a DASH client running on a user device such as an HMD). Thisinformation may include aspects such as the number of availableviewpoints, the position and range of each viewpoint, and when videodata are available for a viewpoint. Moreover, since most omnidirectionalmedia presentations are experienced through a head-mounted display, asudden change in viewpoint may feel unnatural to a viewer who isimmersed in a virtual environment. It is therefore preferable to supportviewpoint transition effects that would provide a smooth transition whenthe user changes their viewpoint. These transitions can also be used bycontent producers to guide the users experience.

Grouping Viewpoint Media Components.

In some embodiments, media samples for omnidirectional media contentwith multiple viewpoints are stored in a number of tracks within thecontainer file. A video player playing or streaming the content operatesto identify which tracks belong to which viewpoint. To enable this, amapping is provided between the media tracks within the file and theviewpoint to which they belong. In some embodiments, this mapping issignaled at the media container (file format) level. In someembodiments, this mapping is signaled at the transport protocol-level(DASH).

Media Container-Level Signaling (File Format).

In ISO/IEC 14496-12 (ISO BMFF), a TrackGroupBox is defined to enablegrouping a number of tracks in the container file that share a certaincharacteristic or have a particular relationship. The track group boxcontains zero or more boxes, and the particular characteristic or therelationship is indicated by the box type of the contained boxes. Thecontained boxes include an identifier, which can be used to conclude thetracks belonging to the same track group. The tracks that contain thesame type of a contained box within the TrackGroupBox and have the sameidentifier value within these contained boxes belong to the same trackgroup.

aligned(8) class TrackGroupBox extends Box(‘trgr’) { }

A track group type is defined extending a TrackGroupTypeBox whichcontains a track_group_id identifier and a track_group_type which storesa four-character code identifying the group type. The pair oftrack_group_id and track_group_type identifies a track group within thefile.

To group a number of media tracks that belong a single viewpointtogether, in some embodiments, a new group type (ViewpointGroupTypeBox)is defined as follows:

aligned(8) class ViewpointGroupTypeBox extends TrackGroupTypeBox(‘vpgr’){ // additional viewpoint data can be defined here }

In some embodiments, media have a viewpointGroupTypeBox within theTrackGroupBox, and tracks that belong to the same viewpoint have thesame value of track_group_id in the respective ViewpointGroupTypeBox. A3DoF+ omnidirectional media player can therefore identify availableviewpoints by parsing each track in the container and checking thenumber of unique track_group_id values within the ViewpointGroupTypeBoxfor each track.

Transport Protocol-Level Signaling (DASH).

The OMAF standard defines delivery-related interfaces for DASH. In someembodiments, information related to the different viewpoints is signaledin the media presentation descriptor. In DASH, each media component isrepresented by an AdaptationSet element in the MPD. In some embodiments,AdaptationSet elements belonging to the same viewpoint are grouped byeither defining an additional attribute to the AdaptationSet element oradding a descriptor to the AdaptationSet where a viewpoint identifier isprovided.

A number of descriptors are defined in the MPEG-DASH standard. Theseinclude a SupplementalProperty descriptor which can be used by the mediapresentation author to express that the descriptor contains supplementalinformation that may be used by the DASH client for optimizedprocessing. The semantics of the signaled information are specific tothe scheme employed, which is identified by the @schemeIdUri attribute.In the present disclosure, a number of new XML elements and attributesare described for signaling viewpoint-related information. The newelements can either be defined in the same namespace as the one definedin the latest version of the OMAF standard (urn:mpeg:mpegI:omaf:2017) orin a separate new namespace (urn:mpeg:mpegI:omaf:2019) to distinguishbetween OMAF v1 features and OMAF v2 features. For the sake ofexplanation, the namespace (urn:mpeg:mpegI:omaf:2017) is used in theremainder of this document.

To identify and describe the viewpoint to which a media componentbelongs, embodiments are described in which a SupplementalPropertyelement is added with a @schemeIdUri attribute equal to“urn:mpeg:mpegI:omaf:2017:ovp”. Such a descriptor is referred to hereinas an OMAF viewpoint (OVP) descriptor. In some embodiments, at most oneOVP descriptor may be present at the adaptation set level. The OVPdescriptor may have a @viewpoint_id attribute with a value representinga unique viewpoint identifier. Examples of semantics for @viewpoint_idare given in Table 1. AdaptationSet elements with the same @viewpoint_idvalue may be recognized by the player as belonging to the sameviewpoint.

TABLE 1 Semantics of omaf:@viewpoint_id attribute Attribute for OVPdescriptor Use Data type Description omaf:@viewpoint_id M xs:stringSpecifies a unique viewpoint within the scene. For ISO Base Media FileFormat Segments, the value of @viewpoint_id shall be equal totrack_group_id in the ViewpointGroupTypeBox in sample entries of theInitialization Segment.Signaling Viewpoint Information.

In order for a player to identify the attributes belonging to differentviewpoints (e.g., spatial relationship between viewpoints, availabilityof the viewpoint, etc.), in some methods described herein, additionalmetadata describing the viewpoint is signaled in the container file (andin the MPD file in the case of streaming). Examples of viewpointattributes that are signaled in some embodiments include viewpointposition, viewpoint effective range, viewpoint type, and viewpointavailability. Viewpoint position specifies the position of the viewpointwithin the 3D space of the captured scene. A viewpoint's effective rangeis the distance from the viewpoint within which objects can be renderedwith a certain level of quality. The certain level of quality may be,for example, a minimum level of quality, a quality level exceeding aknown quality threshold, a guaranteed level of quality, or a level ofquality approved by or deemed acceptable to the provider of theomnidirectional media content. For example, an object that is within theeffective range would be of sufficient size in the rendered image toprovide a resolution that provides good quality and ensures anacceptable viewing experience for a user. The effective range of aviewpoint depends on the characteristics of the capturing device (e.g.,camera sensor resolution, field-of-view, etc.). The effective range maybe determined at least in part by the camera lens density, representingthe number of lenses integrated into a 360-degree video camera.

FIG. 5 shows an example of the effective range of different cameras. Inthis example, viewpoints 502, 504, 506, 508, 510, 512, 514, and 516 areillustrated along with dotted circles 503, 505, 507, 509, 511, 513, 515,and 517 indicating effective ranges of the respective viewpoints. Theomnidirectional cameras located at viewpoints 502 and 510 may containmore lenses to cover a broad area, and so the effective ranges ofviewpoints 502 and 510 may be able to cover the penalty areas 520, 522as illustrated in FIG. 5 . The cameras along the side of the field inthis example may have fewer lenses so the effective range of theseviewpoints (504, 506, 508, 512, 514, 516) may be smaller than that ofviewpoints 502 and 510. In general, an omnidirectional camera which hasmore lenses, more component cameras, or which has higher qualitycomponent cameras (e.g. component cameras with high quality optics, highresolution, and/or high frame rate) may be associated with a highereffective range.

In another embodiment, the viewpoint effective range may be determinedat least in part by camera lens parameters such as focal length,aperture, depth of field and focus distance, etc. The effective rangemay define a minimum range and maximum range, with the effective rangebeing between the minimum range and maximum range without stitchingerror.

A viewpoint may be categorized as a real viewpoint or as a virtualviewpoint. A real viewpoint is a viewpoint where an actual capturingdevice was placed to capture the scene from that position of theviewpoint. A virtual viewpoint refers to a viewpoint where the renderingof viewports at that position calls for performing additionalprocessing, such as view synthesis, which may make use of auxiliaryinformation and/or video data from one or more other (e.g. real)viewpoints.

Viewpoint availability specifies at what time(s) during the presentationare media data available for the viewpoint.

The user's interaction with a viewport scene such as zoom-in or zoom-outmay be supported within the effective range. A virtual viewpoint mayonly be identified within the effective range of one or multiplecameras. The effective range may also be used as reference to generate atransition path. For example, a transition from viewpoint A to viewpointB may involve multiple transition viewpoints if the effective range ofthese viewpoints covers the transition path.

Media Container-Level Signaling of Viewpoint Information.

In ISO BMFF, viewpoint-related information for static viewpoints can besignaled in the ‘meta’ box (Meta Box) at the file level. The ‘meta’ boxholds static metadata and contains only one mandatory box (HandlerBox)which declares the structure or format of the Meta Box. In someembodiments, for OMAF v2 metadata, the four-character code ‘omv2’ isused for the handler_type value in the ‘hdlr’ box. To identify theavailable viewpoints in the file, some embodiments use a box calledOMAFViewpointListBox which contains a list of OMAFViewpointInfoBoxinstances. Each OMAFViewpointInfoBox holds information about a certainviewpoint. An example of syntax of the OMAFViewpointListBox is asfollows.

Box Type: ‘ovpl’ Container: MetaBox Mandatory: No Quantity: Zero or onealigned(8) class OMAFViewpointListBox extends Box(‘ovpl’) { unsignedint(16) num_viewpoints; OMAFViewpointInfoBox viewpoints[ ]; }

An example of semantics for OMAFViewpointListBox is as follows:

num_viewpoints indicates the number of viewpoints in the media file.viewpoints is a list of OMAFViewpointInfoBox instances.

An example of a syntax of OMAFViewpointInfoBox is given below.

Box Type: ‘ovpi’ Container: OMAFViewpointListBox Mandatory: No Quantity:Zero or more aligned(8) class OMAFViewpointInfoBox extends Box(‘ovpi’) {unsigned int(16) viewpoint_id; bit(1) effective_range_flag; bit(1)virtual_viewpoint_flag; bit(1) dynamic_position_flag; bit(5) reserved;if (effective_range_flag == 1) { unsigned int(32) effective_range; }unsigned int(32) num_availability_intervals;OMAFViewpointPositionGlobalBox( ); // optionalOMAFViewpointPositionCartesianBox( ); // optionalOMAFViewpointAvailabilityIntervalBox availability_intervals[ ]; Boxother_boxes[ ];}

An example of semantics of OMAFViewpointInfoBox is as follows:

-   -   viewpoint_id is a unique identifier for the viewpoint.    -   virtual_viewpoint_flag indicates whether the viewpoint is a        viewpoint is a virtual viewpoint (with no capturing device        placed at the position of the viewpoint) or a captured        viewpoint. Information needed to generate the virtual viewpoint        is signaled in the OMAFVirtualViewpointConfigBox.    -   dynamic_position_flag indicates whether the position is static        or dynamic. If this flag is set, the position of the viewpoint        is provided using a timed-metadata track. Otherwise, the        position is indicated by an OMAFViewpointPositionGlobalBox        and/or an OMAFViewpointPositionCartesianBox in this        OMAFViewpointlnfoBox.    -   effective_range is the radius defining a volumetric sphere        centered at the viewpoint for which the viewpoint provides        rendering at a certain quality (e.g. a minimum level of quality,        a quality level exceeding a known quality threshold, a        guaranteed level of quality, or a level of quality approved by        or deemed acceptable to the provider of the omnidirectional        media content).    -   num_availability_intervals indicates the number of time        intervals during which this viewpoint is available.    -   availability_intervals is a list of        OMAFViewpointAvailabilityIntervalBox instances.

In some embodiments, when the viewpoint position in space changes overtime, position information is signaled using a timed-metadata track.Timed metadata tracks are tracks within the media container (ISO BMFF)file where the samples represent dynamic metadata information. Fordynamic viewpoint position information, some embodiments use atimed-metadata track with the sample entry type ‘vpps’. The sample entryfor this track may be as follows.

aligned(8) class OMAFDynamicViewpointSampleEntry extendsMetadataSampleEntry(‘vpps’) { unsigned int(16) viewpoint_id; unsignedint(3) coordinate_system_type; bit(5) reserved; }

An example of semantics for the OMAFDynamicViewpointSampleEntry is asfollows.

-   -   viewpoint_id is the identifier of the viewpoint for which the        samples of this timed-metadata track define the position.    -   coordinate_system_type indicates the coordinate system used for        defining the position of the viewpoint.

In some embodiments, samples for the viewpoint position metadata trackhave the following structure.

aligned(8) class OMAFViewpointPositionSample { if(coordinate_system_type == 1) { ViewpointPositionGlobalStruct( ); } elseif (coordinate_system_type == 2) { ViewpointPositionCartesianStruct( );} }

The sample format may be dependent on the coordinate system type definedin the sample entry of the timed-metadata track.ViewpointPositionGlobalStruct and ViewpointPositionCartesianStruct aredescribed in further detail below.

Transport Protocol-Level Signaling of Viewpoint Information.

To identify and describe the set of viewpoints that are available in amedia presentation, some embodiments include a SupplementaryPropertydescriptor at the Period level. This descriptor may have a @schemeIdUriequal to “urn:mpeg:mpegI:omaf:2017:ovl” and is referred to herein as anOMAF viewpoint list (OVL) descriptor. In some embodiments, at most oneOVL descriptor may be present at the Period level. The OVL descriptormay contain at least one ovp element. An ovp element has an @idattribute with a value representing a unique viewpoint identifier andmay contain sub-elements with information about the viewpoint.

Table 2 lists examples of elements and attributes used for signalingviewpoint information in the MPD file for DASH clients. More details aregiven below.

TABLE 2 Semantics of elements and attributes of example OVL descriptor.Elements and attributes for OVL descriptor Use Data type Description ovp1 . . . N Container element whose attributes and elements specify thecharacteristics of a viewpoint. ovp@id M xs:string A unique identifierfor the viewpoint. ovp@effectiveRange O xs:unsignedInt Indicates therange from within the viewpoint (in metres) for which the viewpoint isable to provide a certain quality of rendering (e.g. a minimum level ofquality, a quality level exceeding a known quality threshold, aguaranteed level of quality, or a level of quality approved by or deemedacceptable to the provider of the omnidirectional media content).ovp@virtual OD xs:boolean A flag indicating whether the viewpoint is areal viewpoint with a capturing device at the viewpoint position (value‘false’) or a virtual viewpoint (value ‘true’) which requires otherreference viewpoints and/or additional auxiliary information to renderthe scene for the viewpoint position. If not present, the default valueis ‘false’. ovp@synthesisMethod CM xs:unsignedByte Indicates whichindicates the method used to generate the virtual viewpoint. Thisattribute shall only be present when @virtual has the value ‘true’.ovp@refViewpointIds CM xs:string A comma separated list of viewpointidentifiers. The viewpoints in this list are used as references by thesynthesis process that generates this (virtual) viewpoint. Thisattribute shall be present only when @virtual has the value ‘true’.ovp@dynamicPosition OD xs:boolean A flag indicating whether the positionof the viewpoint changes over time or is static. If not present, thedefault value is ‘false’. ovp:position CM omaf:viewpointPositionType Acontainer element whose elements identify the position of the viewpointwithin the world of the presentation. This element is present only when@dynamicPosition = ‘false’ ovp:position:global O omaf: An element whoseattributes identify the viewpointGlobalPositionType position of theviewpoint in terms of geolocation. ovp:position: M xs:double Longitudevalue of the viewpoints global@longitude position measured in degrees.ovp:position: M xs:double Latitude value of the viewpoint's positionglobal@latitude measured in degrees. ovp:position: O xs:double Altitudevalue of the viewpoint's position global@altitude measured in degrees.ovp:position:Cartesian O omaf: An element whose attributes identify theviewpointCartesianPositionType position of the viewpoint in terms ofCartesian coordinates. ovp:position: M xs:int The x-axis coordinate.cartesian@x ovp:position: M xs:int The y-axis coordinate. cartesian@yovp:position: OD xs:int The z-axis coordinate. If not present, thecartesian@z default value is zero. ovp:availability 1 . . . Nomaf:viewpointAvailabilityType An element whose attributes define a timeinterval during which the viewpoint is available in the presentation.ovp:availability@start M xs:unsignedLong Presentation time which definesthe start of a time interval during which the viewpoint is available.The viewpoint may not be available before this time. The value of thepresentation time in seconds is the division of the value of thisattribute and the value of the @timescale attribute.ovp:availability@end O xs:unsignedLong Presentation time which definesthe end of a time interval during which the viewpoint is available. Theviewpoint may no longer be available after this time. The value of thepresentation time in seconds is the division of the value of thisattribute and the value of the @timescale attribute. If not present, theviewpoint continues to be available until the end of the presentation.

In Table 2 and other tables in the present disclosure, elements arebold; attributes are non-bold and preceded with an @. “M” indicatesthat, in the particular embodiment shown in the table, the attribute ismandatory, “0” indicates that, in the particular embodiment shown in thetable, the attribute is optional, “OD” indicates that, in the particularembodiment shown in the table, the attribute is optional with a defaultvalue, “CM” indicates that, in the particular embodiment shown in thetable, the attribute is conditionally mandatory. <minOccurs> . . .<maxOccurs> (N=unbounded).

The data types for various elements and attributes are as defined in theXML schema. An XML schema for ovp is provided in the section “XML Schemafor DASH signaling,” below.

Viewpoint Position.

“Real” viewpoints correspond to 360° video cameras that are placed atdifferent positions to capture the scene from different vantage points.In some embodiments, viewpoints may represent views from virtualpositions. The virtual positions may represent points which are notassociated with the location of a physical camera. The virtual positionsmay represent points from which synthetic content may be rendered, orpoints from which content captured by one or more cameras at other(real) viewpoints may be transformed, processed or combined in order tosynthesize a virtual view. To provide the player with useful informationon the camera setup used to capture the scene and their layout, thespatial relationship between the viewpoints in some embodiments issignaled by providing the position of each viewpoint. Positioninformation may be represented in different ways in differentembodiments. In some embodiments, global geolocation coordinates similarto the ones used by GPS systems may be used to identify to location ofthe camera/viewpoint. Alternatively, the Cartesian coordinate system maybe used for positioning.

Media Container-Level Signaling of Viewpoint Position.

Described herein are two examples of boxes that may be used to identifythe position of the viewpoint when present in the OMAFViewpointInfoBox,namely OMAFViewpointPositionGlobalBox andOMAFViewpointPositionCartesianBox. In some embodiments, these boxes areoptional. An example syntax of the proposed position boxes is givenbelow. Additional boxes may also be introduced to provide positioninformation based on other coordinate systems.

Box Type: ‘vpgl’ Container: OMAFViewpointInfoBox Mandatory: No Quantity:Zero or one aligned(8) class ViewpointPositionGlobalStruct( ) {  signedint(32) longitude;  signed int(32) latitude;  signed int(32) altitude; }aligned(8) class OMAFViewpointPositionGlobalBox extends Box(‘vpgl’) { ViewpointPositionGlobalStruct( ); }

In some embodiments, double-precision or floating point types are usedfor the longitude, latitude, and/or altitude values.

Box Type: ‘vpcr’ Container: OMAFViewpointInfoBox Mandatory: No Quantity:Zero or one aligned(8) class ViewpointPositionCartesianStruct( ) { signed int(32) x;  signed int(32) y;  signed int(32) z; } aligned(8)class OMAFViewpointPositionCartesianBox extends Box(‘vpcr’) { ViewpointPositionCartesianStruct( ); }Transport Protocol-Level Signaling of Viewpoint Position.

To signal the position of a viewpoint, in some embodiments, anovp:position element may be added to the ovp element. This element mayinclude an ovp:position:global element and/or an ovp:position:cartesianelement. In some embodiments, at most one of each of these elements ispresent within an ovp:position element. Attributes of theovp:position:global element provide the position of the viewpoint interms of global geolocation coordinates in units of degrees. In someembodiments, the ovp:position:global element has three attributes:@longitude, @latitude, and @altitude. In some embodiments, the @altitudeattribute is optional and may not be present. Attributes of theovp:position:catersian attribute provide the position of the viewpointin terms of Cartesian coordinates. In some embodiments, three attributesare defined for the ovp:position:cartesian element: @x, @y, and @z,where only @z is optional.

Viewpoint Availability.

In some cases, a viewpoint may not be available for the entire durationof the media presentation. Therefore, in some embodiments, theavailability of a viewpoint is signaled before the media samples forthat viewpoint are processed. This enables a player to only process thesamples for the tracks belonging to a specific viewpoint when theviewpoint is available.

Changes in viewpoint availability over time are illustrated in FIG. 6 .At time t1, only viewpoints 601, 602, 603, and 604 are available. Laterduring the presentation at time t2, a penalty shot is given to one ofthe teams and most of the players are close to the goal on the rightside. At that point in time, two additional viewpoints 605 and 606 aremade available to the user until time t3. The time interval between t2and t3 is an availability interval for viewpoints 605 and 606. Usingviewpoint availability information (e.g. received from a server), aplayer or streaming client operates to indicate to the user theavailability of additional viewpoints at time t2 during playback, e.g.,using the UI shown in FIG. 3 or FIG. 4 . At the start of an availabilityinterval, the player may present options to the user to switch to any orall of the viewpoints which are available (e.g. are newly available)during the availability interval. As shown in FIG. 6 , the user may begiven the options to switch to viewpoints 605 or 606 starting from timet2. At the end of an availability interval, the player may removeoptions to switch to viewpoints which are no longer available after theavailability interval ends. In some embodiments, if the user is still atone of these viewpoints when the availability interval ends (e.g. attime t3 as illustrated in FIG. 6 ), the user may be returned to theviewpoint at which the user was prior to switching to the viewpointwhich is no longer available (e.g. viewpoint 605 or 6066 as illustratedin FIG. 6 ). In some embodiments, viewpoint availability intervals mayalso be signaled for virtual viewpoints. However, the availability ofthese viewpoints is dependent on the availability of other referenceviewpoints as well as any auxiliary information used to support therendering of the virtual viewpoint.

Media Container-Level Signaling of Viewpoint Availability.

In some embodiments, a box (OMAFViewpointAvailaibilityIntervalBox) isintroduced to signal availability intervals. Zero or more instances ofthis box may be present in an OMAFViewpointInfoBox. When noOMAFViewpointAvailaibilityIntervalBox instances are present for aviewpoint, this indicates that the viewpoint is available for the entireduration of the presentation.

Box Type: ‘vpai’ Container: OMAFViewpointInfoBox Mandatory: No Quantity:Zero or more aligned(8) class OMAFViewpointAvailabilityIntervalBoxextends Box(‘vpai’) {  bit(1) open_interval_flag;  bit(7) reserved; unsigned int(64) start_time; // mandatory  unsigned int(64) end_time; }

An example of semantics for OMAFViewpointAvailabilityIntervalBox is asfollows:

-   -   open_inverval_flag a flag indicating the availability interval        is an open interval (value 1) where the viewpoint is available        from start_time until the end of the presentation, or a closed        interval (value 0). If the flag is set (value 1), no end_time        field is present in this box.    -   start_time the presentation time at which the viewpoint is        available (corresponds to the composition time for the first        sample in the interval).    -   end_time the presentation time after which the viewpoint is no        longer available (corresponds to the composition time of the        last sample in the interval).        Transport Protocol-Level Signaling of Viewpoint Availability.

In some embodiments, to signal the availability of a viewpoint in theMPD file, one or more ovp:availability elements may be added to aninstance of the ovp element. This element signifies an availabilityperiod and has two attributes, @start and @end, indicating thepresentation time at which the viewpoint is available and thepresentation time of the last sample of the availability interval,respectively.

Virtual Viewpoints.

In some embodiments, virtual viewpoints are generated using anomnidirectional virtual view synthesis process. In some embodiments,this process makes use of one or more input (reference) viewpoints andtheir associated depth maps and additional metadata describing thetranslation vectors between the input viewpoints positions and thevirtual viewpoint position. In some such embodiments, each pixel of theinput omnidirectional viewpoints is mapped to a position in the virtualviewpoint sphere by mapping the pixels of the equirectangular frames ofthe reference viewpoints to points in 3D space and then projecting themback at the target virtual viewpoint. One such view synthesis process isdescribed in greater detail in “Extended VSRS for 360-degree video”,MPEG121, Gwangju, Korea, January 2018, m41990, and is illustrated inFIG. 8 . In the example of FIG. 8 , a point 802 has a position describedby angular coordinates (φ,θ) and depth z with respect to an inputviewpoint 804. In the generation of a virtual viewpoint 806, which isdisplaced from input viewpoint 804 by a vector (T_(x), T_(y), T_(z)),angular coordinates (φ′,θ′) are found for the point 802 with respect tothe virtual viewpoint 806. The displacement vector (T_(x), T_(y), T_(z))may be determined based on viewpoint positions signaled in a containerfile, a manifest, a timed-metadata track, or otherwise.

Various techniques may be used to generate virtual viewpoint indifferent embodiments. Virtual viewpoint frames synthesized fromdifferent reference viewpoints may then be merged together using ablending process to generate the final equirectangular frame at thevirtual viewpoint. Holes appearing the final frame due to occlusions atthe reference viewpoint may be processed using an inpainting and holefilling step.

A virtual viewpoint is a non-captured viewpoint. Viewports can berendered at a virtual viewpoint using video data from other viewpointsand/or other supplementary information. In some embodiments, theinformation used to render the scene from a virtual viewpoint issignaled in an OMAFVirtualViewpointConfigBox that is present in theOMAFViewpointInfoBox when the virtual_viewpoint flag is set. In someembodiments, the OMAFirtualViewpointConfigBox may be defined as follows.

Box Type: ‘vvpc’ Container: OMAFViewpointInfoBox Mandatory: No Quantity:Zero or more aligned(8) class OMAFVirtualViewpointConfigBox extendsBox(‘vvpc’) {  unsigned int(5) synthesis_method;  unsigned int(3)num_reference_viewpoints;  unsigned int(16) reference_viewpoints[ ];  //optional boxes but no fields }

Examples of semantics for the OMAFVirtualViewpointConfigBox fields aregiven below.

-   -   synthesis_method indicates which synthesis method is used to        generate the virtual viewpoint. The value of synthesis_method        may be an index to a listed table of view synthesis methods. For        example: depth-image-based rendering, image-warping-based        synthesis, etc.    -   num_reference_viewpoints indicates the number of viewpoints that        will be used as references in the synthesis of the virtual        viewpoint.    -   reference_viewpoints is a list of the viewpoint ids which are        used as references when synthesizing a viewport for this        viewpoint.

In another embodiment, the identifiers of the tracks containinginformation needed for the synthesis process are directly signalled in avirtual viewpoint configuration box, which may be implemented asfollows.

aligned(8) class OMAFVirtualViewpointConfigBox extends Box(‘vvpc’) { unsigned int(5) synthesis_method;  unsigned int(3)num_reference_tracks;  unsigned int(16) reference_track_ids[ ];  //optional boxes but no fields }

An example of semantics of the OMAFVirtualViewpointConfigBox fields forthis embodiment is as follows.

-   -   synthesis_method indicates which synthesis method is used to        generate the virtual viewpoint. The value of synthesis_method is        the index to a listed table of view synthesis methods. For        example: depth-image-based rendering, image-warping-based        synthesis, etc.    -   num_reference_tracks indicates the number of tracks within the        container file that will be used as references in the synthesis        of the virtual viewpoint.    -   reference_track_ids is a list of track identifiers for the        tracks used in the synthesis of viewports for this viewpoint.        Signaling Viewpoint Groups.

In large-scale events such as the FIFA World Cup, a number of events maybe running in parallel at different venues or locations. For example, anumber of games may take place in different stadiums, possibly indifferent cities. In some embodiments, viewpoints can be grouped basedon the geolocation of the event/venue. In some embodiments, aViewpointGroupStruct structure is used to store information about agroup of viewpoints within the media container file. An example ofsyntax of this structure is as follows.

aligned(8) class ViewpointGroupStruct( ) {  unsigned int(8)viewpoint_group_id;  signed int(32) longitude;  signed int(32) latitude; unsigned int(8) num_viewpoints;  unsigned int(16) viewpoint_ids[ ]; string viewpoint_group_name; }

An example of semantics of the fields of ViewpointGroupStruct is asfollows.

-   -   viewpoint_group_id is a unique id that identifies the viewpoint        group.    -   longitude is the longitude coordinate of the geolocation of the        event/venue where the viewpoints are located.    -   latitude is the latitude coordinate of the geolocation of the        event/venue where the viewpoints are located.    -   num_viewpoints is the number of viewpoints within the viewpoint        group.    -   viewpoint_ids is an array with the ids of the viewpoints that        are part of the viewpoint group.    -   viewpoint_group_name is a string with a name describing the        group.

To signal the available viewpoint groups within the media containerfile, an OMAFViewpointGroupsBox may be added to the MetaBox in the ISOBMFF container file. An example of the syntax of an OMAFViewpointGroupsBox is given below.

Box Type: ‘ovpg’ Container: MetaBox Mandatory: No Quantity: Zero or onealigned(8) class OMAFViewpointGroupsBox extends Box(‘ovpg’) {  unsignedint(8) num_viewpoint_groups;  ViewpointGroupStruct viewpoint_groups[ ];}

An example of semantics for fields of this box is follows:

-   -   num_viewpoint_groups is the number of viewpoint groups.    -   viewpoint_groups is an array of ViewpointGroupStruct instances,        providing information about each viewpoint group.

For transport-protocol-level signaling (e.g. DASH), to signal theviewpoint groups available in a media presentation, an ovg element maybe defined and signaled in the OVL descriptor described above. The OVLdescriptor may contain one or more ovg elements. An ovg element has an@id attribute with a value representing a unique viewpoint groupidentifier and other attributes that describe the group. Table 3 listsattributes of an example of an ovg element.

TABLE 3 Semantics of the attributes of an example omaf:ovg element.Elements and attributes for OVL descriptor Use Data type Description @idM xs:string Container element whose attributes and elements specify thecharacteristics of a viewpoint. @name O xs:string A name for the group.@longitude M xs:double The longitude of the geolocation of theevent/venue at which the viewpoint group is located, measured indegrees. @latitude M xs:double The latitude of the geolocation of theevent/venue at which the viewpoint group is located, measured indegrees. @viewpointIds M xs:string A comma-separated list of theidentifiers of the viewpoints that belong to the group.Signaling Viewpoint Transition Effects.

Disclosed herein are the following examples of types of transitions:basic transitions, viewpoint path transitions, and auxiliary informationtransitions. Basic transitions are predefined transitions that can beused when switching from one viewpoint to another. An example of such atransition is the fade-to-black transition, where the rendered viewgradually fades out to black then fades in to a frame from the newviewpoint. A viewpoint path transition enables content producers tospecify a path that the player may follow across other viewpoints whenswitching to the target viewpoint. Auxiliary information transitions aretransitions which rely on auxiliary information that the contentproducer provides in separate tracks. For example, an auxiliary trackmay contain depth information which can be used to render intermediatevirtual views as the viewport moves from the first viewpoint to thetarget viewpoint.

In some embodiments, transitions may be based on the rendering ofintermediate virtual views. This can be done using a view synthesisprocess such as depth-image-based rendering (DIBR), described forexample in C. Fehn, “Depth-image-based rendering (DIBR), compression,and transmission for a new approach on 3D-TV,” in SPIE StereoscopicDisplays and Virtual Reality Systems XI, vol. 5291, May 2004, pp.93-104. DIBR uses depth information to project the pixels in a 2D planeto their position in 3D space and re-project them back to another plane.Since no capturing devices (e.g., no 360-degree cameras) are present atthese intermediate viewpoints, they are referred to herein as virtualviewpoints. The number of intermediate virtual viewpoints renderedbetween the source and destination viewpoints determines the smoothnessof the transition and depends on the capabilities of the player/deviceand the availability of auxiliary information for these intermediateviewpoints.

FIG. 7 illustrates an embodiment using virtual viewpoints. In theexample of FIG. 7 , only viewpoints 702, 704, 706, 708, 710, 712, 714,716 are viewpoints with capturing devices and the remaining intermediateviewpoints (703, 705, 707, 709, 711, 713, 715, 717) are virtualviewpoints. Other types of auxiliary information include: point cloudstreams, additional reference frames from nearby viewpoints (to enhancethe quality of virtual views), and occlusion information (to support thehole-filling step in the view synthesis process and improve the qualityof the resulting virtual view at intermediate viewpoints). In someembodiments, point cloud streams are used to enable rendering virtualviews at arbitrary viewpoint positions between the source anddestination viewpoints. In some embodiments, point clouds are renderedusing techniques described in Paul Rosenthal, Lars Linsen, “Image-spacepoint cloud rendering”, in Proceedings of Computer GraphicsInternational, pp. 136-143, 2008.

Media Container-Level Signaling of Viewpoint Transition Effects.

Some embodiments operate to signal transition effects between pairs ofviewpoints in the container file as a list of boxes in a newOMAFViewpointTransitionEffectListBox which can be placed in the Meta Boxat the file level. In some embodiments, at most one instance of this boxis present in the MetaBox. Boxes in OMAFViewpointTransitionEffectListBoxare instances of OMAFViewpointTransitionBox. An example of syntax of thetwo boxes is given below.

Box Type: ‘vptl’ Container: MetaBox Mandatory: No Quantity: Zero or onealigned(8) class OMAFViewpointTransitionEffectListBox extendsBox(‘vptl’) {  OMAFViewpointTransitionBox transitions[ ]; } Box Type:‘vpte’ Container: OMAFViewpointTransitionEffectListBox Mandatory: NoQuantity: One or more aligned(8) class OMAFViewpointTransitionEffectBoxextends Box(‘vpte’) {  unsigned int(16) src_viewpoint_id; // mandatory unsigned int(16) dst_viewpoint_id; // mandatory  unsigned int(8)transition_type; // mandatory  // additional box to specify theparameters of the transition }

An example of semantics for the fields of OMAFViewpointTransitionBox isas follows:

-   -   src_viewpoint_id is the id of the source viewpoint.    -   dst_viewpoint_id is the id of the destination viewpoint.    -   transition_type is an integer identifying the type of        transition. Value 0 indicates a basic transition. Value 1        indicates a viewpoint path transition. Value 2 indicates an        auxiliary information transition. Remaining values are reserved        for future transitions.

In some embodiments, additional boxes related to the specific type oftransition and providing additional information may be present in theOMAFViewpointTransitionBox. An additional box may be defined for each ofthe previously defined transitions types. AnOMAFBasicViewpointTransitionBox is present if the transition_type fieldof an OMAFViewpointTransitionBox is equal to 0. This box contains onlyone field, basic_transition_type, whose value indicates a specifictransition from a set of pre-defined basic transitions. AOMAFPathViewpointTransitionBox is present when the transition_type fieldof OMAFViewpointTransitionBox is equal to 1. This box contains a list ofviewpoint identifiers which the player can follow when the user requestsa transition to the target viewpoint. In some embodiments, a field mayalso be provided to indicate the speed of the transition along the path.A OMAFAuxiliaryInfoViewpointTransitionBox is present when thetransition_type field of OMAFViewpointTransitionBox is equal to 2. Thisbox contains two fields: a type field specifying the nature of thetransition (e.g., generating virtual viewpoints), and an aux_track_idproviding a reference to one of the tracks in the file which includestimed auxiliary information used to perform the transition effect.Examples of the syntax of the three aforementioned boxes are givenbelow.

aligned(8) class OMAFBasicViewpointTransitionBox extends Box(‘vptb’) { unsigned int(8) basic_transition_type; { aligned(8) classOMAFPathViewpointTransitionBox extends Box(‘vptp’) {  unsigned int(16)intermediate_viewpoints[ ]; } aligned(8) classOMAFAuxiliaryInfoViewpointTransitionBox extends Box(‘vpta’) {  unsignedint(8) type;  unsigned int(32) aux_track_id; }Transport Protocol-Level Signaling (e.g. DASH) of Viewpoint TransitionEffects.

Viewpoint transition effect information signaled at the container-levelmay also be signaled at the transport protocol level in the manifestfile. If the container file contains viewpoint transition effectinformation, this information preferably matches the informationsignaled in the manifest file. In some embodiments, viewpoint transitioneffects are signaled within an OVL descriptor such as that describedabove. A transition effect between a viewpoint pair may be signaled byan ovp:transition element. In one example, this element has threeattributes: @src, @dst, and @type. These attributes designate the id ofthe source viewpoint, the id of the destination viewpoint, and the typeof the transition effect, respectively. For certain types of transitioneffects, the ovp:transition element may contain child elements providingadditional information used by the client to render these transitions.

Table 4 lists examples of elements and attributes that may be used forsignaling viewpoint transition effects in the MPD file.

TABLE 4 Semantics of elements and attributes of an exampleovp:transition element. Elements and attributes for ovp:transition UseData type Description ovp:transition@src M xs:string The identifier forthe source viewpoint. ovp:transition@dst M xs:string The identifier forthe destination viewpoint. ovp:transition@type M xs:unsignedByteIndicates the type of the transition. Value 0 indicates a basictransition. Value 1 indicates a path transition. Value 2 indicates anauxiliary information transition. Other values are reserved.ovp:transition:basic CM xs:vpBasicTransitionType This element shall onlybe present if ovp:transition@type attribute is set to 0.ovp:transition:basic M xs:unsignedByte Indicates the type of the basictransition effect. @type ovp:transition:path CM xs:vpPathTransitionTypeProvides additional information about the path transition effect. Thiselement shall only be present if ovp:transition@type attribute is setto 1. ovp:transition:path M xs:string A comma separated list of theidentifiers of @viewpoints viewpoints along the path from the sourceviewpoint to the destination viewpoint. ovp:transition:aux CMxs:vpAuxTransitionType Provides additional information about theauxiliary information transition effect. This element shall only bepresent if ovp:transition@type attribute is set to 2. ovp:transition:auxM xs:string A comma-separated list of the ids of auxiliary @auxIdListinformation AdaptationSet elements.Signaling Recommended Projection Format for FoV.

Different projection formats may advantageous within different FoVranges. For example, a rectilinear projection format may work well at afield of view of 90°, but an undesirable stretching effect may bevisible using rectilinear projection at larger fields of view, such as130°. Conversely, projection formats such as a “little planet”stereographic projection or a fisheye projection format may not workwell at a FoV of 90° but may present a reasonable rendering experienceat a higher FoV degree.

In some embodiments, to signal the recommended projection format for arange of device field of view (FoV) values, aOMAFRecommendedProjectionListBox is provided as additional metadatainformation in the ‘meta’ box. This box contains one or moreOMAFRecommendedProjectionBox instances. An OMAFRecommendedProjectionBoxdefines horizontal and vertical FoV ranges and provides a recommendedprojection type for the specified FOV ranges. A player or streamingclient which receives this signaling may determine the size of the fieldof view of the device on which the player or streaming client is running(e.g. it may look up the device's FOV capabilities from a localdatabase, or it may obtain this property through an API call to theoperating system of the HMD). The player or streaming client may comparethis determined field of view size to the FOV ranges defined in theOMAFRecommendedProjectionBoxes in order to determine which of therecommended projection types corresponds to the field of view of thedevice. The player or streaming client may then request content in thedetermined recommended projection format. Examples of the syntax forthese boxes are provided below.

Box Type: ‘orpl’ Container: MetaBox Mandatory: No Quantity: Zero or onealigned(8) class OMAFRecommendedProjectionListBox extends Box(‘orpl’) { OMAFRecommendedProjectionBox recommendations[ ]; { Box Type: ‘orpr’Container: OMAFRecommendedProjectionListBox Mandatory: No Quantity: Oneor more aligned(8) class OMAFRecommendedProjectionBox extendsBox(‘orpr’) {  bit(3) reserved = 0;  unsigned int(5) projection_type; unsigned int(32) min_hor_fov;  unsigned int(32) min_ver_fov;  unsignedint(32) max_hor_fov;  unsigned int(32) max_ver_fov; }

Examples of semantics of the fields OMAFRecommendedProjectionBox are asfollows:

-   -   projection_type indicates the type of the mapping of the        projected picture onto the spherical coordinate system as        specified by the OMAF standard. The value of projection_type may        be the index of a list of rendering projection methods including        rectilinear projection, little planet projection, equidistant        projection, fisheye projection, etc.    -   min_hor_fov and min_ver_fov provide a minimum horizontal and        vertical display field of view, in units of 2⁻¹⁶ degrees.        min_hor_fov may be in the range of 0 to 360*2¹⁶, inclusive.        min_ver_fov may be in the range of 0 to 180*2¹⁶, inclusive.    -   max_hor_fov and max_ver_fov provide a maximum horizontal and        vertical display field of view, in units of 2⁻¹⁶ degrees.        max_hor_fov may be in the range of 0 to 360*2¹⁶, inclusive.        max_ver_fov may be in the range of 0 to 180*2¹⁶, inclusive.

In a case in which the projection format is recommended for a specificFoV, min_hor_fov is equal to max_hor_fov and min_ver_fov is equal tomax_ver_fov.

In another embodiment, the content author or content provider mayprovide information identifying a recommended viewport for the deviceswith different FoV configurations with the suitable projectionrecommendation. Different devices with different FoVs may follow therecommended viewport and use the recommended projection format to renderthe 360 video content.

OMAF describes a recommended viewport information box (RcvpInfoBox) asfollows.

class RcvpInfoBox extends FullBox(‘rvif’, 0, 0) {  unsigned int(8)viewport_type;  string viewport_description; }

The viewport_type specifies the type of the recommended viewport aslisted in Table 5.

TABLE 5 Recommended viewport type. Value Description 0 A recommendedviewport suggested according to the creative intent of the contentauthor or content provider (e.g. the “director's cut) 1 A recommendedviewport selected based on measurements of viewing statistics  2 . . .239 Reserved (for use by future extensions of ISO/IEC 23090-2) 240 . . .255 Unspecified (for use by applications or external specifications)

In some embodiments, an additional type of recommended viewport (whichmay be assigned, e.g., type 2) is used based on the FOV of the renderingdevices. In some embodiments, the viewport_description of RcvpInfoBoxmay be used to indicate the recommended rendering projection method andthe corresponding rendering FOV range. In some embodiments, an optionalbox is added in a RcvpInfoBox based on the viewport_type to indicate theadditional parameters used for the corresponding recommended type. Forexample, OMAFRecommendedProjectionBox may be signaled when the viewporttype is associated with the FOV.

class RcvpInfoBox extends FullBox(‘rvif’, 0, 0) {  unsigned int(8)viewport_type;  string viewport_description;  Box[ ] other_boxes; //optional }

In another embodiment, a recommended viewport may accommodate multiplerecommended types, or sub-types to offer user flexible selection. Forexample, the viewing statistics may be further divided into thestatistics by the measuring period (e.g. weekly, monthly), the geography(countries, cities) or ages (youth, adult). Table 6 illustrates ahierarchical recommendation structure that may be used in someembodiments.

TABLE 6 Director's Cut 2D Rendering General PG-13 R NC-17 VR RenderingGeneral PG-13 R NC-17 Statistics Time Period Today Week Month YearGeographic North America South America Asia Europe

A recursive RcvpInfoBox structure is used in some embodiments to supporta hierarchical recommendation structure. The other_boxes field proposedin a RcvpInfoBox structure may include RcvpInfoBox to specify the subtype as follows.

class RcvpInfoBox extends FullBox(‘rvif’, 0, 0) {  unsigned int(8)viewport_type;  string viewport_description;  RcvpinfoBox( ); //optional; }

A single directors cut recommended viewport may offer multiple tracks,and each may support one or more recommended rendering projectionmethods for a FOV range. A RcvpInfoBox example structure is illustratedbelow. The value of viewport_type of the primary RcvpInfoBox is 0indicating such recommended viewport is per director's cut, and thevalue of viewport_type (e.g. 1) in the secondary RcvpInfoBox mayindicate the track associated with this director's cut recommendedviewport is recommended for the device with particular rendering FOV.One or more instances of OMAFRecommendedProjectionBox may be signaled toprovide recommended projection method(s) for the corresponding FOVrange.

RcvpInfoBox{  viewport_type = 0; // recommended director's cut RcvpInfoBox {   Viewport_type = 1; // recommeded for device FOV  OMAFRecommendedProjectionBox( ); // projection method 1  OMAFRecommendedProjectionBox( ); // projection method 2  viewport_description;  }  viewport_description; }

In a DASH MPD, the SupplementalProperty and/or EssentialPropertydescriptors with @schemeIdUri equal to “urn:mpeg:dash:crd” may be usedto provide a content recommendation description (CRD). The @value of theSupplementalProperty or EssentialProperty elements using the CRD schememay be implemented as a comma separated list of values for CRDparameters as shown in Table 7.

TABLE 7 EssentialProperty@value and/or SupplementalProperty@valueattributes for an example CRD scheme. EssentialProperty@value orSupplementalProperty@value parameter Use Description recommendation_typeM Non-negative integer in decimal representation providing the primaryrecommendation type such as director's cut or statistic measurement.sub_type O A comma separated list of non-negative integer in decimalrepresentation expressing the hierarchical sub-type of therecommendations. For director's cut recommendation type, the sub-typecan be the film rating type, or rendering device capabilities (e.g. 2Ddisplay, VR display, or stereoscopic display). Multiple sub-types can belisted in a comma separated list to indicate hierarchical sub-typestructure. When not present, the Representation associated to thisdescriptor does not have a sub-type. content_desciption O Stringrepresentation providing additional recommendation information. Forexample, the recommended FOV range or preferred projection method may beincluded here for the director's cut recommendation for VR display sothat the end user can identify the appropriate Object associated to thisdescriptor.XML Schema for DASH Signaling.

An example of an XML schema for DASH signaling that may be used in someembodiments is the following:

<?xml version=“1.0” encoding=“UTF-8”?> <xs:schemaxmlns:xs=“http://www.w3.org/2001/XMLSchema” targetNamespace=“urn:mpeg:mpegI:omaf:2017” xmlns:omaf=“urn:mpeg:mpegI:omaf:2017”  elementFormDefault=“qualified”? <xs:element name=“ovp” type=“omaf:viewpointType”/>  <xs:elementname=“ovg” type=“omaf:viewpointGroupType”/>  <xs:complexTypename=“viewpointType”?   <xs:attribute name=“id” type=“xs:string”use=“required” />   <xs:attribute name=“effective_range”type=“xs:unsignedInt” use=“optional” />   <xs:attribute name=“virtual”type=“xs:boolean” use=“optional” de-fault=“-false” />   <xs:attributename=“synthesisMethod” type=“xs:unsignedByte” use=“optional” />  <xs:attribute name=“refViewpointIds” type=“xs:boolean” use=“optional”/>   <xs:attribute name=“dynamicPosition” type=“xs:boolean”use=“optional” default=“false” />   <xs:element name=“position”type=“omaf:viewpointPositionType” minOccurs=“0” maxOccurs=“1”/>  <xs:element name=“availability” type=“omaf:viewpointAvailabilityType”maxOccurs=“unbounded” />   <xs:element name=“transition”type=“omaf:vpTransitionType” minOccurs=“0” maxOccurs=“unbounded” /> </xs:complexType>  <xs:complexType name=“viewpointPositionType”>  <xs:element name=“global” type=“omaf:viewpointGlobalPositionType”maxOccurs=“1” />   <xs:element name=“cartesian”type=“omaf:viewpointCartesianPositionType” maxOccurs=“1” /> </xs:complexType>  <xs:complexType name=“viewpointGlobalPositionType”use=“optional” maxOccurs=“1”>   <xs:attribute name=“longitude”type=“xs:double” use=“required” />   <xs:attribute name=“latitude”type=“xs:double” use=“required” />   <xs:attribute name=“altitude”type=“xs:double” use=“optional” default=“0” />  </xs:complexType> <xs:complexType name=“viewpointCartesianPositionType” use=“optional”maxOccurs=“1”>   <xs:attribute name=“x” type=“xs:int” use=“required” />  <xs:attribute name=“y” type=“xs:int” use=“required” />   <xs:attributename=“z” type=“xs:int” use=“optional” default=“0” />  </xs:complexType> <xs:complexType name=“viewpointAvailabilityType” use=“optional”maxOccurs=“unbounded”>   <xs:attribute name=“start”type=“xs:unsignedLong” use=“required” />   <xs:attribute name=“end”type=“xs:unsignedLong” use=“optional” />  </xs:complexType> <xs:complexType name=“vpTransitionType” use=“optional”maxOccurs=“unbounded”>   <xs:attribute name=“src” type=“xs:string”use=“required” />   <xs:attribute name=“dst” type=“xs:string”use=“required” />   <xs:attribute name=“type” type=“xs:unsignedByte”use=“required” />   <xs:element name=“omaf:vpBasicTransitionType”use=“optional” maxOccurs=“1” />   <xs:elementname=“omaf:vpPathTransitionType” use=“optional” maxOccurs=“1” />  <xs:element name=“omaf:vpAuxTransitionType” use=“optional”maxOccurs=“1” />  </xs:complexType>  <xs:complexTypename=“vpBasicTransitionType”>   <xs:attribute name=“type”type=“unsignedByte” use=“required” />  </xs:complexType> <xs:complextType name=“vpPathTransitionType”>   <xs:attributename=“viewpoints” type=“xs:string” use=“required” />  </xs:complextType> <xs:complexType name=“vpAuxTransitionType”>   <xs:attributename=“auxIdList” type=“xs:string” use=“required” />  </xs:complexType> <xs:complexType name=“viewpointGroupType”>   <xs:attribute name=“id”type=“xs:string” use=“required” />   <xs:attribute name=“name”type=“xs:string” use=“optional” />   <xs:attribute name=“longitude”type=“xs:double” use=“required” />   <xs:attribute name=“latitude”type=“xs:double” use=“required” />   <xs:attribute name=“viewpointIds”type=“xs:string” use=“required” />  </xs:complexType> </xs:schema>Additional Embodiments.

In some embodiments, a method includes: receiving at least first360-degree video data representing a view from a first viewpoint andsecond 360-degree video data representing a view from a secondviewpoint; and generating a container file (e.g. an ISO Base Media FileFormat file) for at least the first video data and the second videodata. In the container file: the first video data is organized into afirst set of tracks and the second video data is organized in a secondset of tracks; each of the tracks in the first set of tracks includes afirst track-group identifier associated with the first viewpoint; andeach of the tracks in the second set of tracks includes a secondtrack-group identifier associated with the second viewpoint.

In some such embodiments, each of the tracks in the first set of tracksincludes a respective instance of a viewpoint-group-type box thatcontains the first track-group identifier; and each of the tracks in thesecond set of tracks includes a respective instance of aviewpoint-group-type box that contains the second track-groupidentifier.

In some embodiments where the container file is organized in ahierarchical box structure, and the container file includes aviewpoint-list box that identifies at least a firstviewpoint-information box and a second viewpoint-information box, thefirst viewpoint-information box includes at least (i) the firsttrack-group identifier and (ii) an indication of time intervals forwhich video from the first viewpoint is available; and the secondviewpoint-information box includes at least (i) the second track-groupidentifier and (ii) an indication of time intervals for which video fromthe second viewpoint is available. The indications of time intervals maybe lists of instances of respective viewpoint availability intervalboxes.

In some embodiments, where the container file is organized in ahierarchical box structure, and where the container file includes aviewpoint-list box identifying at least a first viewpoint-informationbox and a second viewpoint-information box: the firstviewpoint-information box includes at least (i) the first track-groupidentifier and (ii) an indication of a position of the first viewpoint;and the second viewpoint-information box includes at least (i) thesecond track-group identifier and (ii) an indication of a position ofthe second viewpoint. The indication of position may include Cartesiancoordinates or latitude and longitude coordinates.

In some embodiments where the container file is organized in ahierarchical box structure, and where the container file includes aviewpoint-list box identifying at least a first viewpoint-informationbox and a second viewpoint-information box: the firstviewpoint-information box includes at least (i) the first track-groupidentifier and (ii) an indication of an effective range of the firstviewpoint; and the second viewpoint-information box includes at least(i) the second track-group identifier and (ii) an indication of aneffective range of the second viewpoint.

In some embodiments where the container file is organized in ahierarchical box structure, and the container file includes atransition-effect-list box identifying at least one transition-effectbox, each transition-effect box includes: an identifier of a sourceviewpoint; an identifier of a destination viewpoint; and an identifierof a transition type. The identifier of the transition type may identifya basic transition or a viewpoint path transition. Where the identifierof the transition type identifies a path-viewpoint-transition box, thepath-viewpoint-transition box may include a list of viewpointidentifiers. Where the identifier of the transition type identifies anauxiliary-information-viewpoint-transition box, theauxiliary-information-viewpoint-transition box may include a trackidentifier.

In some embodiments, where the container file is organized in ahierarchical box structure including a meta box: the meta box identifiesat least one recommended-projection-list box; and eachrecommended-projection-list box includes information identifying (i) aprojection type and (ii) a corresponding field-of-view range. Theinformation identifying the corresponding field-of-view range mayinclude: a minimum horizontal field of view angle; a maximum horizontalfield of view angle; a minimum vertical field of view angle; and amaximum vertical field of view angle.

Some embodiments include a non-transitory computer storage mediumstoring a container file generated according to any of the methodsdescribed herein.

In some embodiments a method includes: receiving at least first360-degree video data representing a view from a first viewpoint andsecond 360-degree video data representing a view from a secondviewpoint; and generating a manifest, such as an MPEG-DASH MPD. In themanifest: at least one stream in a first set of streams is identified,each stream in the first set representing at least a portion of thefirst video data; at least one stream in a second set of streams isidentified, each stream in the second set representing at least aportion of the second video data; each of the streams in the first setis associated in the manifest with a first viewpoint identifier and;each of the streams in the second set is associated in the manifest witha second viewpoint identifier.

In some such embodiments, each of the streams in the first set isassociated in the manifest with a respective adaptation set that has thefirst viewpoint identifier as an attribute; and each of the streams inthe second set is associated in the manifest with a respectiveadaptation set that has the second viewpoint identifier as an attribute.The attribute may be a @viewpoint_id attribute.

In some embodiments, each of the streams in the first set is associatedin the manifest with a respective adaptation set that has the firstviewpoint identifier in a first descriptor; and each of the streams inthe second set is associated in the manifest with a respectiveadaptation set that has the second viewpoint identifier in a seconddescriptor. The first and second descriptors may be SupplementalPropertydescriptors.

In some embodiments, the manifest includes an attribute indicating aneffective range for each of the viewpoints. In some embodiments, themanifest includes an attribute indicating a position for each of theviewpoints. The attribute indicating position may include cartesiancoordinates or latitude and longitude coordinates. In some embodiments,the manifest includes, for each viewpoint, information indicating atleast one time period during which video for the respective viewpoint isavailable.

In some embodiments, the first video data and second video data arereceived in a container file (such as an ISO Base Media File Formatfile) in which: the first video data is organized into a first set oftracks and the second video data is organized in a second set of tracks;each of the tracks in the first set of tracks includes a firsttrack-group identifier associated with the first viewpoint; and each ofthe tracks in the second set of tracks includes a second track-groupidentifier associated with the second viewpoint. The viewpointidentifiers used in the manifest are equal to the respective track-groupidentifiers in the container file.

In some embodiments, a method includes: receiving a manifest identifyinga plurality of 360-degree video streams, the manifest including, foreach identified stream, information identifying a viewpoint location ofthe respective stream; retrieving and displaying a first video streamidentified in the manifest; and overlaying on the display of the firstvideo stream a user interface element indicating the viewpoint locationof a second video stream identified in the manifest. In someembodiments, the method includes, in response to selection of the userinterface element, retrieving and displaying the second video stream.

In some embodiments, where the manifest further includes informationidentifying an effective range of at least one of the identifiedstreams, the method further includes displaying an indication of theeffective range. In some embodiments, where the manifest furtherincludes information identifying a period of availability of the secondvideo stream, the user interface element is displayed only during theperiod of availability.

In some embodiments, the manifest includes information identifying atransition type for a transition from the first video stream to thesecond video stream. In response to selection of the user interfaceelement, the method includes: presenting a transition having theidentified transition type; and retrieving and displaying the secondvideo stream, the second video stream being displayed after thepresentation of the transition.

In some embodiments, where the manifest further includes informationidentifying a location of at least one virtual viewpoint, the methodfurther includes, in response to selection of the virtual viewpoint,synthesizing a view from the virtual viewpoint and displaying thesynthesized view.

In some embodiments, a method includes: receiving a manifest (anMPEG-DASH MPD) identifying a plurality of 360-degree video streams, themanifest including information identifying a respective projectionformat of each of the video streams, the manifest further includinginformation identifying a respective range of field-of-view sizes foreach of the projection formats; determining a field-of-view size fordisplay; selecting at least one of the video streams such that thedetermined field-of-view size is within the identified range offield-of-view sizes for the projection format of the selected videostreams; and retrieving at least one of the selected video streams anddisplaying the retrieved video stream with the determined field-of-viewsize.

Further embodiments include a system comprising a processor and anon-transitory computer-readable medium storing instructions operativewhen executed on the processor to perform any of the methods describedherein.

Note that various hardware elements of one or more of the describedembodiments are referred to as “modules” that carry out (i.e., perform,execute, and the like) various functions that are described herein inconnection with the respective modules. As used herein, a moduleincludes hardware (e.g., one or more processors, one or moremicroprocessors, one or more microcontrollers, one or more microchips,one or more application-specific integrated circuits (ASICs), one ormore field programmable gate arrays (FPGAs), one or more memory devices)deemed suitable by those of skill in the relevant art for a givenimplementation. Each described module may also include instructionsexecutable for carrying out the one or more functions described as beingcarried out by the respective module, and it is noted that thoseinstructions could take the form of or include hardware (i.e.,hardwired) instructions, firmware instructions, software instructions,and/or the like, and may be stored in any suitable non-transitorycomputer-readable medium or media, such as commonly referred to as RAM,ROM, etc.

Although features and elements are described above in particularcombinations, one of ordinary skill in the art will appreciate that eachfeature or element can be used alone or in any combination with theother features and elements. In addition, the methods described hereinmay be implemented in a computer program, software, or firmwareincorporated in a computer-readable medium for execution by a computeror processor. Examples of computer-readable storage media include, butare not limited to, a read only memory (ROM), a random access memory(RAM), a register, cache memory, semiconductor memory devices, magneticmedia such as internal hard disks and removable disks, magneto-opticalmedia, and optical media such as CD-ROM disks, and digital versatiledisks (DVDs). A processor in association with software may be used toimplement a radio frequency transceiver for use in a WTRU, UE, terminal,base station, RNC, or any host computer.

What is claimed:
 1. A method comprising: receiving, from a server, amanifest including information identifying omnidirectional videoscaptured from respective viewpoints, the manifest includes informationidentifying a first omnidirectional video captured from a firstviewpoint and a second omnidirectional video captured from a secondviewpoint; receiving, from the server, information identifyingtransition effects associated with respective pairs of source anddestination viewpoints, including receiving information identifying afirst transition effect including one or more viewpoint identifiersassociated with the first viewpoint as the source viewpoint and thesecond viewpoint as the destination viewpoint; and transitioning fromthe first omnidirectional video to the second omnidirectional videousing one or more omnidirectional videos identified in the manifest andcaptured from respective viewpoints identified by the one or moreviewpoint identifiers included in the first transition effect.
 2. Themethod of claim 1, wherein the first transition effect is afade-to-black effect.
 3. The method of claim 1, wherein the informationidentifying the first transition effect identifies a speed of transitionalong a path across the viewpoints used for the transitioning from thefirst omnidirectional video to the second omnidirectional video.
 4. Themethod of claim 1, further comprising receiving, for at least one of thefirst and second viewpoints, information identifying an effective rangeof the respective viewpoint.
 5. The method of claim 1, wherein theinformation identifying the first transition effect identifies auxiliaryinformation applied to render intermediate views used for thetransitioning from the first viewpoint to the second viewpoint.
 6. Themethod of claim 1, further comprising displaying to a user a userinterface, wherein the user interface allows a user to select the secondomnidirectional video based on the viewpoint position of the secondomnidirectional video.
 7. The method of claim 6, further comprisingdisplaying the second omnidirectional video to the user in response touser selection of the second omnidirectional video.
 8. The method ofclaim 6, wherein displaying the user interface comprises: displaying thefirst omnidirectional video to the user; and displaying an indication ofthe second omnidirectional video at a location in the firstomnidirectional video corresponding to the second viewpoint of thesecond omnidirectional video.
 9. The method of claim 8, wherein theindication of the second omnidirectional video has a size based on adistance from the first viewpoint to the second viewpoint.
 10. Themethod of claim 1, further comprising, before the transitioning from thefirst omnidirectional video to the second omnidirectional video: causingdisplay of the first omnidirectional video; and causing display of anicon at a location in the first omnidirectional video corresponding tothe second viewpoint position of the second omnidirectional video,wherein the icon has a size based on a distance from the first viewpointposition to the second viewpoint position.
 11. The method of claim 10,further comprising: in response to user selection of the icon,performing the transitioning from the first omnidirectional video to thesecond omnidirectional video.
 12. The method of claim 10, wherein thesize of the icon is smaller for greater distances between the firstviewpoint position and the second viewpoint position to provide a deptheffect for the second viewpoint position.
 13. The method of claim 10,further comprising receiving, for at least one of the first and secondviewpoints, information identifying an effective range of the respectiveviewpoint.
 14. A system comprising a processor configured to perform atleast: receiving, from a server, a manifest including informationidentifying omnidirectional videos captured from respective viewpoints,the manifest includes information identifying a first omnidirectionalvideo captured from a first viewpoint and a second omnidirectional videocaptured from a second viewpoint; receiving, from the server,information identifying transition effects associated with respectivepairs of source and destination viewpoints, including receivinginformation identifying a first transition effect including one or moreviewpoint identifiers associated with the first viewpoint as the sourceviewpoint and the second viewpoint as the destination viewpoint; andtransitioning from the first omnidirectional video to the secondomnidirectional video using one or more omnidirectional videosidentified in the manifest and captured from respective viewpointsidentified by the one or more viewpoint identifiers included in thefirst transition effect.
 15. The system of claim 14, wherein the firsttransition effect is a fade-to-black effect.
 16. The system of claim 14,wherein the information identifying the first transition effectidentifies a speed of transition along a path across the viewpoints usedfor the transitioning from the first omnidirectional video to the secondomnidirectional video.
 17. The system of claim 14, wherein theinformation identifying the first transition effect identifies auxiliaryinformation applied to render intermediate views used for thetransitioning from the first viewpoint to the second viewpoint.
 18. Thesystem of claim 14, further comprising displaying to a user a userinterface, wherein the user interface allows a user to select the secondomnidirectional video based on the viewpoint position of the secondomnidirectional video.
 19. The system of claim 18, further comprisingdisplaying the second omnidirectional video to the user in response touser selection of the second omnidirectional video.
 20. The system ofclaim 18, wherein displaying the user interface comprises: displayingthe first omnidirectional video to the user; and displaying anindication of the second omnidirectional video at a location in thefirst omnidirectional video corresponding to the second viewpoint of thesecond omnidirectional video.
 21. The system of claim 20, wherein theindication of the second omnidirectional video has a size based on adistance from the first viewpoint to the second viewpoint.